Description

Own the design, development, and optimization of scalable data engineering solutions on Azure, including ETL/ELT pipelines and medallion architectures.

This role is on-site.

Responsibilities

• Design and build scalable data ingestion and transformation workflows using Azure Databricks, PySpark, and Delta Lake.
• Orchestrate complex data pipelines and integrate Azure services (ADLS, Azure SQL, Event Hub, Synapse) via Azure Data Factory.
• Implement CI/CD processes using Git and Azure DevOps, ensuring code quality through reviews and standardized branching strategies.
• Optimize PySpark jobs for performance and cost efficiency, managing compute environments and clusters.
• Collaborate with Data Architects to define design patterns and document solution diagrams and runbooks.

Required Skills

• 5+ years of experience in data engineering with a Bachelor's degree.
• Strong proficiency in PySpark (DataFrames, Spark SQL) for data processing and optimization.
• Hands-on experience with Azure Databricks, including notebooks, jobs, and Delta Lake medallion architecture.
• Deep knowledge of Azure Data Factory (ADF) for pipeline triggers, execution, and integration runtime.
• Practical experience with Data Lake Storage (ADLS Gen2), including partitioning and security configurations.
• Proficiency in SQL for writing and optimizing complex queries.
• Experience implementing CI/CD pipelines using Git and Azure DevOps.

Preferred Skills

• Experience with Azure Synapse Analytics and Event Hub integration.
• Knowledge of data governance, security, and access control best practices.

Key Skills

Education

Bachelor's degree