Description

You will design and implement data pipelines using Python to manage transformations, error handling, and logging across Azure services.

This role is on-site.

Responsibilities

• Build and manage ETL processes to read from diverse sources, including databases and streaming data.
• Load and manage data within Databricks-managed Delta tables.
• Evaluate and execute complex data transformations using PySpark.
• Develop and maintain data pipelines to ensure efficient data flow across Azure services.
• Lead data engineering projects and optimize large-scale data infrastructures.

Required Skills

• 6+ years of experience in data engineering with a focus on building pipelines.
• Strong proficiency in Python, including classes, objects, methods, dictionaries, loops, Wheel files, and packages.
• Extensive experience with Databricks and Apache Spark.
• Hands-on expertise with PySpark and Scala.
• Deep knowledge of Azure Data Factory, Azure Synapse Analytics, Azure SQL Database, and Azure Data Lake.
• Solid understanding of ETL processes and data modeling.
• Proficiency in Unix shell scripting and scheduling tools.

Preferred Skills

• Experience with large-scale data infrastructure optimization.

Key Skills

Education

Any Graduate