Description

Design and maintain data pipelines using Python and Azure services.

Responsibilities

• Build and optimize ETL workflows in Python to extract, transform, and load data from diverse sources.
• Develop and manage data pipelines using Azure Data Factory.
• Perform data transformations and analytics using Databricks.
• Implement data governance frameworks and ensure data quality standards and compliance.
• Configure monitoring, alerts, and notifications for batch job execution.

Required Skills

• 5+ years of experience in data engineering.
• Strong proficiency in Python programming for ETL processes.
• Hands-on experience with Microsoft Azure cloud platform.
• Practical knowledge of Azure Data Factory, Databricks, Azure Storage, and Azure Virtual Machines.
• Understanding of data governance principles and data quality management.
• Ability to design efficient data pipelines and implement data controls.

Preferred Skills

• Experience implementing alerting mechanisms for batch jobs.
• Knowledge of Azure architecture best practices.

Key Skills

Education

Any Gradute