Description

You will build and maintain scalable data pipelines and lakehouse infrastructure on Microsoft Azure. Your work supports efficient extraction, transformation, and loading of data across batch and real-time workloads.

Responsibilities

• Implement and manage Medallion Architecture (Bronze → Silver → Gold) using Azure Data Factory, Databricks-PySpark, and Azure SQL Database.
• Build robust ADF pipelines with ForEach, Lookup, Copy Activity, and Data Flows, ensuring incremental loading via watermark or CDC.
• Perform root cause analysis on pipeline failures and data quality issues to resolve SLA breaches and identify platform improvements.
• Implement SLA-based data quality checks (freshness, completeness, row count) and monitor via Azure Monitor and ADF diagnostic logs.
• Manage Git-based workflows and CI/CD pipeline promotion across Dev/Test/Prod using Azure DevOps or GitHub Actions.

Required Skills

• 5+ years building big data pipelines using Azure Data Factory, PySpark, and SQL.
• Hands-on experience implementing Medallion Architecture (Bronze/Silver/Gold).
• Experience with Delta Lake: ACID transactions, incremental loading, schema evolution, and partitioning strategies.
• Proficiency in Python and PySpark for data transformation, pipeline automation, and large-scale distributed processing.
• Strong SQL skills including window functions, CTEs, and query optimization across relational and lakehouse engines.
• Working knowledge of Azure Foundational Services: ADLS Gen2, Azure SQL Database, Azure Blob Storage, Azure Key Vault, Azure Monitor / Log Analytics, Azure Event Hubs, and Microsoft Fabric Lakehouse.
• Understanding of Azure Active Directory / Entra ID (RBAC, Service Principals).
• Solid understanding of dimensional modeling (Star Schema, SCD Types 1/2/3) and lakehouse vs. data warehouse trade-offs.
• Experience delivering data platforms for executive-level reporting via Power BI semantic models.

Preferred Skills

• Experience with Microsoft Fabric (Lakehouse, Notebooks, OneLake, Fabric Pipelines) in active migration or greenfield projects.
• Experience with real-time / streaming workloads using Azure Event Hubs or Structured Streaming in PySpark.

Key Skills

Education

Any Gradute