Description

You will design, build, and maintain production-grade data pipelines on Azure, focusing on reliability, performance, and automated testing.

This role is hybrid.

Responsibilities

• Develop and optimize ETL/ELT pipelines using Python and PySpark, ensuring modular code, comprehensive testing, and high code coverage.
• Build and manage Azure Data Factory (ADF) pipelines with advanced parameterization, triggers, and robust error handling.
• Operate Databricks notebooks and jobs, implementing Delta Lake patterns and optimizing distributed transformations.
• Provision and manage Azure infrastructure using Pulumi, including ADLS Gen2, Azure SQL, and resource groups.
• Implement CI/CD workflows for data services, including unit/integration test automation and performance tuning for complex SQL queries.

Required Skills

• 5+ years of experience in data engineering with Python and SQL.
• Strong proficiency in PySpark for distributed data processing and performance optimization.
• Hands-on experience with Azure Data Factory (ADF) for pipeline orchestration.
• Experience with Databricks and Delta Lake patterns.
• Knowledge of CI/CD practices, including testing automation and code coverage.
• Familiarity with Infrastructure as Code tools like Pulumi.
• Bachelor's degree in a technical field.

Preferred Skills

• Experience with Azure OpenAI integration.
• Advanced SQL performance tuning skills.

Key Skills

Education

Bachelor's degree