Description

Design, build, and maintain end-to-end data pipelines for large-scale datasets. You will own the full lifecycle from ingestion to publishing, ensuring reliability and performance.

Responsibilities

• Develop production-grade Python modules with robust logging, error handling, and testing.
• Create and manage Azure Data Factory pipelines, including triggers, parameterization, and monitoring.
• Optimize Databricks clusters, jobs, and workflows for performance and Delta Lake architecture.
• Provision and manage cloud infrastructure using Infrastructure as Code tools.
• Collaborate with cross-functional teams to troubleshoot data issues and support production systems.

Required Skills

• 5+ years of experience in Data Engineering with strong ETL/ELT processes.
• 3+ years of experience with PySpark and advanced SQL, including complex queries, joins, CTEs, and performance tuning.
• 2+ years of hands-on Python development in production environments.
• Experience with Azure Data Factory for pipeline development and monitoring.
• Hands-on experience with Azure services such as ADLS Gen2 and Azure SQL.
• Solid understanding of data pipeline architecture, incremental loads, and data validation.

Key Skills

Education

Bachelor's degree