Description

You will build and manage end-to-end data pipelines and ETL/ELT workflows within the Azure ecosystem.

Responsibilities

• Design, develop, and maintain data pipelines and ETL/ELT workflows using PySpark and Python.
• Implement and optimize large-scale workloads in Azure Databricks, including cluster configuration, autoscaling, and governance.
• Author Infrastructure-as-Code using ARM templates to deploy repeatable Azure environments.
• Configure and operate Azure services including Synapse, Key Vault, VMSS, Function Apps, and Azure Container Apps.
• Manage CI/CD pipelines and source control using GitLab and Azure DevOps.
• Provide on-call support for production issues and deployments.

Required Skills

• 10+ years of professional experience in data engineering.
• Advanced proficiency in Python and PySpark.
• Expertise in writing SQL for analytics and ETL processes.
• Hands-on experience with Azure Databricks cluster management, job scheduling, and workspace governance.
• Deep knowledge of Azure services: Storage Account, Synapse, Key Vault, VMSS, Function Apps, Web Apps, and Log Analytics.
• Experience with container services such as Azure Container Apps and container instances.
• Experience with Azure networking concepts and secure network integration.
• Proficiency with GitLab and Azure DevOps for CI/CD and release management.
• Bachelor's degree.

Preferred Skills

• Experience working within Agile/Scrum frameworks.
• Understanding of data governance, lineage, and compliance frameworks.

Key Skills

Education

Bachelor's degree