Description

Design and implement large-scale migration pipelines from on-prem systems to Azure Data Lake Storage (ADLS) and Azure Data Factory (ADF). You will own the end-to-end data engineering lifecycle, including ingestion, transformation, and analytics, using Azure Databricks and Delta Lake.

Responsibilities

• Build and maintain scalable data pipelines using Azure Databricks and Scala.
• Implement infrastructure-as-code for pipeline management using Terraform.
• Develop and optimize data workflows in Azure Data Factory (ADF) for ingestion and transformation.
• Perform data analysis to troubleshoot issues and support stakeholder analytics needs.
• Ensure data quality, logging, and traceability for audit and compliance.

Required Skills

• 8-10 years of experience in data engineering or data warehouse development.
• Hands-on expertise with Azure Databricks and Delta Lake.
• Strong proficiency in Scala for data processing and UI development within pipelines.
• Experience building and managing pipelines with Azure Data Factory (ADF).
• Proficiency with Terraform for infrastructure provisioning.
• Deep understanding of SQL, including complex joins, duplicate record identification, and data analysis.
• Experience working in agile development environments and sprint delivery.

Key Skills

Education

Any Graduate