Description

You will design and build end-to-end data pipelines for a forecasting platform, adhering to rigorous engineering standards.

Responsibilities

• Develop, test, and document scalable data pipelines.
• Build and evolve data integrations, including APIs, to handle increasing data volume and complexity.
• Establish and follow data governance processes to ensure data integrity and security.
• Design, implement, and automate deployment of distributed systems for streaming event processing.
• Collaborate with analytics teams to improve data models feeding business intelligence tools.

Required Skills

• 7+ years of relevant work experience in data engineering or related fields.
• Strong proficiency in Python and object-oriented programming.
• Expertise in PySpark and advanced SQL querying on complex datasets.
• Strong experience with Databricks.
• Proven experience developing integrations across multiple systems and APIs.
• Experience with Azure technologies, specifically Azure Data Lake, ADF, and Azure DevOps.
• Familiarity with Agile software development methodologies.
• Experience designing and operating SQL/NoSQL database systems.

Preferred Skills

• Experience with Snowflake.
• Knowledge of Kafka.

Key Skills

Education

Any Graduate