Description

You will lead the design and implementation of scalable data pipelines and architectures on Azure Databricks.

Responsibilities

• Design and implement ETL/ELT pipelines to move data from Azure SQL into Databricks Delta Lake.
• Optimize ETL/ELT workflows for efficient data processing, storage, and retrieval.
• Troubleshoot and fine-tune Apache Spark jobs for performance and cost efficiency.
• Lead the migration of Azure SQL workloads to Azure Databricks.
• Ensure data governance, security, and compliance within Azure environments.

Required Skills

• 5+ years of professional experience in Data Engineering.
• Expert proficiency with Azure Databricks.
• Strong command of Apache Spark for large-scale data processing.
• Experience with Delta Lake architecture.
• Proficiency in SQL optimization and query tuning.
• Experience designing scalable data pipelines.
• Knowledge of Azure-native services.

Key Skills

Education

Any Graduate