Description

You will design and lead the end-to-end architecture of a scalable data platform on Azure, ensuring low-latency, high-reliability pipelines for batch and real-time data.

This role is hybrid.

Responsibilities

• Design and implement end-to-end data platform architecture using Databricks, Delta Lake, Snowflake, and Azure services.
• Build scalable ETL/ELT pipelines and real-time streaming processes using Kafka or Azure Event Hubs with a 15-minute SLA.
• Establish data quality frameworks, monitoring, and governance standards to ensure data completeness and consistency.
• Optimize pipeline performance and storage for large-scale datasets, implementing partitioning, indexing, and caching strategies.
• Lead and mentor a team of data engineers while collaborating with product managers and BI teams to deliver trusted data layers for Power BI and Microsoft Fabric.

Required Skills

• 5+ years of experience in data engineering and platform architecture.
• Expertise in Databricks, Spark, and Delta Lake for large-scale data processing.
• Strong experience with Snowflake or similar cloud data warehouses.
• Hands-on proficiency with streaming technologies such as Kafka or Azure Event Hubs.
• Strong programming skills in Python and SQL.
• Experience with data modeling, ETL/ELT design, and data quality frameworks.
• Familiarity with Power BI and Microsoft Fabric for semantic models and analytics.
• Bachelor's degree in Computer Science, Engineering, or related field.

Preferred Skills

• Experience with Azure ecosystem tools including ADF, ADLS, and AKS.
• Familiarity with CI/CD and DevOps practices for data platforms.

Key Skills

Education

Bachelor's degree