Description

You build and maintain scalable data pipelines and lakehouse infrastructure on Microsoft Azure to support batch and real-time workloads.

This role is hybrid.

Responsibilities

• Implement and manage Medallion Architecture (Bronze, Silver, Gold) using Azure Data Factory, Databricks PySpark, and Azure SQL Database.
• Develop robust ETL/ELT pipelines with Delta Lake, handling ACID transactions, incremental loading, and schema evolution.
• Ensure SLA-adherent data quality standards, performing root cause analysis on pipeline failures and data quality issues.
• Deploy and manage Azure Foundational Services including ADLS Gen2, Azure Key Vault, Azure Monitor, and Event Hubs.
• Collaborate with BI teams to feed Gold-layer data into Power BI, managing DirectQuery vs. Import mode trade-offs.

Required Skills

• 5+ years building big data pipelines using Azure Data Factory, PySpark, and SQL.
• Hands-on experience with Azure SQL Database, Databricks Unity Catalog, and Delta Lake.
• Proficiency in Python and PySpark for data transformation and large-scale distributed processing.
• Strong SQL skills, including window functions, CTEs, and query optimization.
• Experience implementing dimensional modeling (Star Schema, SCD Types 1/2/3).
• Knowledge of ADF pipeline patterns: ForEach, Lookup, Copy Activity, Data Flows, and error handling.
• Experience with Git-based workflows and CI/CD using Azure DevOps or GitHub Actions.
• Ability to manage concurrent projects and communicate status to technical and non-technical stakeholders.

Preferred Skills

• Experience with Microsoft Fabric (Lakehouse, Notebooks, OneLake, Fabric Pipelines).
• Experience with real-time streaming workloads using Azure Event Hubs or Structured Streaming.

Key Skills

Education

Bachelor's degree