Description

Lead the architecture and execution of enterprise-grade, cloud-native data pipelines and lakehouse solutions using Azure Data Services and Azure Databricks.

This role is hybrid.

Responsibilities

• Lead the design and implementation of modern data platform architectures, specifically Medallion Architecture (Bronze, Silver, Gold layers) using Azure Data Services and Databricks Delta Lake.
• Build and optimize end-to-end, scalable, and resilient batch and streaming ETL/ELT pipelines using PySpark, SQL, Azure Data Factory (ADF), and Databricks Workflows.
• Perform deep-dive Spark tuning, cluster policy optimization, partitioning, liquid clustering, and query optimization to ensure cost-efficiency and high execution speeds.

Required Skills

• 7+ years of experience in data engineering roles.
• Strong proficiency in Azure Data Services and Azure Databricks.
• Hands-on experience with PySpark and SQL for data processing and transformation.
• Expertise in Azure Data Factory (ADF) for pipeline orchestration.
• Demonstrated ability to design and implement Medallion Architecture patterns.
• Proven track record in performance tuning and cost optimization of large-scale data workloads.

Preferred Skills

• Experience with liquid clustering and advanced Databricks optimization techniques.
• Background in designing resilient streaming data solutions.

Key Skills

Education

Any Graduate