Description

Architect scalable Databricks Lakehouse solutions on Azure, lead legacy ETL migrations, and mentor data engineering squads.

This role is on-site.

Responsibilities

• Design and own secure Databricks Lakehouse architectures using Delta Lake, Unity Catalog, and ADLS Gen2.
• Build and review PySpark, Spark SQL, and Delta Live Tables pipelines while setting engineering standards.
• Drive large-scale migrations from legacy systems like DataStage to Databricks and Azure Data Factory.
• Optimize cluster performance, cost, and FinOps practices while embedding governance via Unity Catalog and Purview.
• Lead technical mentoring, define CI/CD best practices in Azure DevOps, and partner with senior stakeholders.

Required Skills

• 12+ years in data engineering or platform architecture with 4+ years of hands-on Databricks delivery.
• Expert-level PySpark, Spark SQL, Delta Lake, Delta Live Tables, and Unity Catalog.
• Strong Azure data stack: Azure Data Factory, ADLS Gen2, Azure Key Vault, and Azure networking.
• Proven experience migrating complex legacy ETL (DataStage, Informatica, Teradata) to cloud lakehouse.
• Deep knowledge of medallion architecture, dimensional modeling, SCD handling, and data quality frameworks.
• Engineering rigor: CI/CD, Git version control, automated testing, and production support of critical pipelines.
• Ability to lead engineers and engage senior client stakeholders while contributing code and designs.

Preferred Skills

• Databricks Certified Data Engineer Professional or Solutions Architect, plus Azure DP-203/AZ-305.
• Experience in retail, supply chain, or financial services data domains.
• Familiarity with IBM DataStage, DB2, Oracle, and legacy on-prem ETL estates.

Key Skills

Education

Bachelor's degree