Description

You will architect and maintain scalable data pipelines on Azure Databricks, translating business requirements into robust data models for BI and AI initiatives.

Responsibilities

• Architect data transformation workflows using dbt (models, tests, macros) within the Databricks Medallion Architecture (Bronze/Silver/Gold).
• Build and optimize high-throughput data pipelines using Azure Databricks (PySpark, SQL), dbt, Airflow, and Azure Data Lake Storage (ADLS).
• Design star and snowflake schema data models to support BI reporting, ad-hoc analysis, and machine learning projects.
• Implement CI/CD for dbt via GitHub and Azure DevOps, ensuring rigorous testing, documentation, and data quality checks using Unity Catalog.
• Optimize Spark jobs, SQL queries, and dbt models for performance and cost efficiency while mentoring junior engineers.

Required Skills

• 8-10 years of experience in data engineering roles.
• Expert-level proficiency with Azure Databricks, PySpark, and Spark SQL.
• Advanced experience with dbt for data transformation and modeling.
• Strong proficiency in Python and SQL.
• Hands-on experience with Apache Airflow for workflow orchestration.
• Deep understanding of Medallion Architecture and Kimball data modeling methodologies.
• Proficiency with Git/GitHub, CI/CD practices, and Azure DevOps.
• Familiarity with Unity Catalog or similar data governance and cataloging tools.

Preferred Skills

• Experience creating packages and working with SDKs.
• Background in domain-specific analytics such as Retail or Manufacturing.
• Databricks certification.

Key Skills

Education

Any Gradute