Description

You will architect and drive the modernization of the data platform by leveraging Databricks capabilities.

Responsibilities

• Lead the migration of data pipelines and workflows from Azure Data Factory to Databricks, focusing on performance optimization and minimal downtime.
• Architect and implement scalable data processing frameworks using Apache Spark, PySpark, Delta Lake, and Databricks notebooks.
• Define and implement data governance best practices, covering lineage, quality, cataloging, and access control.
• Optimize data pipelines for cost-efficiency, scalability, and performance across the Databricks environment.
• Establish and maintain CI/CD pipelines and DevOps processes for Databricks environments.

Required Skills

• 8+ years in data engineering/architecture, with 2+ years specifically in Databricks.
• Proven experience migrating ADF pipelines to Databricks, including orchestration and transformation.
• Strong expertise in Apache Spark, PySpark, SQL, and Delta Lake.
• Experience with the Azure data ecosystem, including ADF, ADLS Gen2, Synapse, and Key Vault.
• Good understanding of data governance frameworks, including metadata management and RBAC.
• Familiarity with CI/CD practices in Databricks using tools like Terraform or ARM templates.
• Experience with performance tuning and cost optimization in Databricks.
• Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related field.
• Strong documentation discipline and problem-solving skills.

Preferred Skills

• Experience working with Unity Catalog in Databricks.
• Knowledge of data quality tools.

Key Skills

Education

Any Graduate