Description

Build and manage end-to-end data pipelines and ETL/ELT workflows within the Azure ecosystem.

This role is on-site.

Responsibilities

• Design, develop, and maintain data pipelines using PySpark and Python.
• Optimize large-scale workloads in Azure Databricks, including cluster configuration and autoscaling.
• Implement Infrastructure-as-Code using ARM templates for repeatable Azure deployments.
• Configure Azure services including Synapse, Key Vault, VMSS, Function Apps, and Container Apps.
• Manage CI/CD pipelines and source control using GitLab and Azure DevOps.

Required Skills

• 10+ years of professional experience in data engineering.
• Advanced proficiency in Python and PySpark.
• Expertise in writing SQL for analytics and ETL processes.
• Hands-on experience with Azure Databricks cluster management, job scheduling, and workspace governance.
• Deep knowledge of Azure services: Storage Account, Synapse, Key Vault, VMSS, Function Apps, Web Apps, and Log Analytics.
• Experience with container services such as Azure Container Apps and container instances.
• Proficiency with GitLab and Azure DevOps for CI/CD and release management.

Preferred Skills

• Experience working within Agile/Scrum frameworks.
• Understanding of data governance, lineage, and compliance frameworks.

Key Skills

Education

Bachelor's degree