Description

Build and optimize scalable ETL/ELT pipelines on Azure using PySpark and Databricks.

This role is on-site.

Responsibilities

• Design and maintain scalable ETL/ELT pipelines using PySpark and Azure Databricks.
• Build and optimize data ingestion frameworks using Azure Data Factory (ADF).
• Develop data transformation pipelines for structured and semi-structured data sources.
• Implement data models within Azure Data Lake and Delta Lake architecture.
• Automate deployments using CI/CD pipelines and Infrastructure as Code.

Required Skills

• 7+ years of experience in Data Engineering.
• Strong expertise in PySpark and Apache Spark.
• Hands-on experience with Azure Databricks and Azure Data Factory (ADF).
• Proficiency in SQL development and query optimization.
• Experience working with Azure Data Lake Storage (ADLS Gen2).
• Knowledge of Delta Lake architecture and data lake concepts.
• Experience with Git, Azure DevOps, and CI/CD pipelines.

Preferred Skills

• Experience with Azure Synapse Analytics.

Key Skills

Education

Bachelor's degree