Description

You will design and build data pipelines to collect, clean, and aggregate data from various sources and APIs.

Responsibilities

• Analyze data sources and APIs to design methods for data collection and ingestion.
• Develop processes to filter and cleanse data for downstream use.
• Write SQL, Hive queries, and APIs to extract data from storage.
• Collaborate with Data Scientists to ensure data is correctly aggregated and prepared.
• Partner with Cloud Architects and DevOps to define cloud architectures and build automated data pipelines.

Required Skills

• 6 to 10 years of relevant data engineering experience.
• Proficiency in Python and PySpark.
• Hands-on experience with Apache Spark and Hadoop.
• Experience with Hive and NoSQL databases.
• Strong understanding of Big Data ecosystems.
• Proficiency with GIT and version control systems.
• Experience in client-facing roles with strong communication skills.
• Domain knowledge in Finance, Banking, Fraud, or Payments.

Preferred Skills

• Experience with Snowflake.
• Cloud platform experience.

Key Skills

Education

Any Graduate