Description

You will design and implement data pipelines using Python to manage transformations, error handling, and logging across Azure services.

Responsibilities

• Build and manage ETL processes to read from diverse sources, including databases and streaming data.
• Load and manage data within Databricks-managed Delta tables.
• Evaluate and execute complex data transformations using PySpark.
• Lead data engineering projects and optimize large-scale data infrastructures.
• Develop and maintain data pipelines to ensure efficient data flow across Azure services.

Required Skills

• 6+ years of experience in data engineering with a focus on building pipelines.
• Strong proficiency in Python, including classes, objects, methods, dictionaries, loops, Wheel files, and packages.
• Extensive experience with Databricks and Apache Spark.
• Hands-on expertise with PySpark and Scala.
• Deep knowledge of Azure Data Factory, Azure Synapse Analytics, Azure SQL Database, and Azure Data Lake.
• Solid understanding of ETL processes and data modeling.
• Experience with Spark, data lakes, cloud warehouses, and storage formats.
• Proficiency in Unix shell scripting and scheduling tools.

Preferred Skills

• Experience with large-scale data infrastructure optimization.

Key Skills

Education

Any Graduate