Description

You will design, develop, and maintain end-to-end data pipelines and ETL/ELT workflows using PySpark and Python. This role is on-site.

Responsibilities

• Migrate and review legacy DataStage code, ensuring functional parity in the new Databricks environment.
• Implement, optimize, and monitor large-scale data processing workloads in Azure Databricks, including cluster configuration and autoscaling.
• Author Infrastructure-as-Code for Azure resources (ARM templates) and deploy consistent, repeatable environments.
• Configure and operate Azure components including Storage Accounts, Synapse, Key Vault, VMSS, and Function Apps.
• Implement secure data access patterns using Azure Active Directory, Managed Identities, and service principals.

Required Skills

• 5+ years of experience with PySpark and Python for data engineering.
• Hands-on experience with Azure Databricks, including cluster management and cost optimization.
• Proficiency in building and maintaining CI/CD pipelines using GitLab and Azure DevOps.
• Strong understanding of ETL/ELT workflows and data integration patterns.
• Experience with Azure Synapse, Key Vault, and Azure networking services.
• Familiarity with Infrastructure-as-Code (ARM templates) and Git version control.
• Ability to implement monitoring, alerting, and governance using Log Analytics.

Preferred Skills

• Experience with Terraform for infrastructure provisioning.
• Familiarity with Agile/Scrum practices and sprint planning.

Key Skills

Education

Bachelor's degree