Description

You will architect and drive the modernization of the data platform using Databricks capabilities.

This role is on-site.

Responsibilities

• Lead the migration of data pipelines and workflows from Azure Data Factory to Databricks.
• Architect and implement scalable data processing frameworks using Apache Spark and Delta Lake.
• Define and implement data governance best practices, including lineage, quality, and access control.
• Optimize data pipelines for cost-efficiency, scalability, and performance.
• Establish and maintain CI/CD pipelines and DevOps processes for Databricks environments.

Required Skills

• 8+ years in data engineering/architecture, with 2+ years specifically in Databricks.
• Proven experience migrating ADF pipelines to Databricks, including orchestration and transformation.
• Strong expertise in Apache Spark, PySpark, SQL, and Delta Lake.
• Experience with the Azure data ecosystem, including ADF, ADLS Gen2, Synapse, and Key Vault.
• Good understanding of data governance frameworks, including metadata management and RBAC.
• Familiarity with CI/CD practices in Databricks using tools like Terraform or ARM templates.
• Experience with performance tuning and cost optimization in Databricks.
• Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related field.

Preferred Skills

• Experience working with Unity Catalog in Databricks.
• Knowledge of data quality tools.

Key Skills

Education

Any Graduate