Description

You will build and maintain scalable data pipelines and lakehouse infrastructure on Microsoft Azure.

This role is hybrid.

Responsibilities

• Implement Medallion Architecture (Bronze/Silver/Gold) using Azure Data Factory, Databricks-PySpark, and Azure SQL Database.
• Build ADF pipelines with ForEach, Lookup, and Copy Activity, ensuring incremental loading via watermark or CDC.
• Perform root cause analysis on pipeline failures and data quality issues to resolve SLA breaches.
• Implement SLA-based data quality checks (freshness, completeness) and monitor via Azure Monitor and ADF diagnostic logs.
• Manage Git-based workflows and CI/CD pipeline promotion across Dev/Test/Prod using Azure DevOps or GitHub Actions.

Required Skills

• 5+ years building big data pipelines using Azure Data Factory, PySpark, and SQL.
• Hands-on experience implementing Medallion Architecture (Bronze/Silver/Gold).
• Experience with Delta Lake: ACID transactions, incremental loading, schema evolution, and partitioning strategies.
• Proficiency in Python and PySpark for data transformation and large-scale distributed processing.
• Strong SQL skills including window functions, CTEs, and query optimization.
• Working knowledge of Azure Foundational Services: ADLS Gen2, Azure SQL Database, Azure Blob Storage, Azure Key Vault, Azure Monitor, and Azure Event Hubs.
• Understanding of Azure Active Directory / Entra ID (RBAC, Service Principals).
• Solid understanding of dimensional modeling (Star Schema, SCD Types 1/2/3) and lakehouse vs. data warehouse trade-offs.
• Experience delivering data platforms for executive-level reporting via Power BI semantic models.

Preferred Skills

• Experience with Microsoft Fabric (Lakehouse, Notebooks, OneLake, Fabric Pipelines) in active migration or greenfield projects.
• Experience with real-time / streaming workloads using Azure Event Hubs or Structured Streaming in PySpark.

Key Skills

Education

Any Gradute