Description

You will design and maintain end-to-end data pipelines and ETL/ELT workflows using PySpark and Python within Azure Databricks.

This role is hybrid.

Responsibilities

• Design, develop, and optimize large-scale data processing workloads in Azure Databricks, including cluster configuration, autoscaling, and governance.
• Build data integration and orchestration solutions using Azure services to meet performance, availability, and security requirements.
• Implement secure data access patterns using Azure Active Directory, Managed Identities, and service principals.
• Author Infrastructure-as-Code for Azure resources using ARM templates and deploy consistent environments.
• Configure and operate Azure components including Synapse, Key Vault, and Container Apps, while implementing monitoring and cost optimization.

Required Skills

• Strong programming skills in Python and PySpark.
• Advanced proficiency in writing SQL for analytics and ETL processes.
• Proven experience building and optimizing complex data pipelines in Azure.
• Hands-on experience with Azure Databricks: cluster management, job scheduling, and workspace governance.
• Strong working knowledge of core Azure services: Storage Account, Synapse, Key Vault, VMSS, Function Apps, Web Apps, and Log Analytics Workspace.
• Experience with container services (ACA, container instances) and containerized data workloads.
• Familiarity with Azure networking concepts and secure network integration.
• Proficient with GitLab and Azure DevOps for CI/CD and source control workflows.
• 10+ years of experience in data engineering roles.

Preferred Skills

• Experience creating Azure infrastructure using ARM templates.
• Familiarity with Agile/Scrum practices and data governance frameworks.

Key Skills

Education

Any Graduate