Description

You will design, build, and maintain end-to-end data pipelines and ETL/ELT workflows using PySpark and Python on Azure Databricks.

This role is on-site.

Responsibilities

• Design and optimize large-scale data processing workloads, including cluster configuration, autoscaling, and governance in Azure Databricks.
• Migrate and validate legacy DataStage code to Databricks, ensuring functional parity and performance.
• Implement secure data access patterns using Azure Active Directory, Managed Identities, and service principals.
• Author Infrastructure-as-Code for Azure resources (ARM templates, Terraform) and manage CI/CD pipelines using GitLab and Azure DevOps.
• Configure and operate Azure services including Storage Accounts, Synapse, Key Vault, Function Apps, and Log Analytics for monitoring and cost optimization.

Required Skills

• 5+ years of experience with PySpark and Python for data engineering.
• Strong proficiency in Azure Databricks, including cluster management and optimization.
• Hands-on experience with Azure DevOps and GitLab for source control and CI/CD.
• Experience with Terraform and Bash scripting for infrastructure automation.
• Knowledge of Azure networking, security (AAD, Managed Identities), and data governance.
• Bachelor's degree in Computer Science or related field.

Key Skills

Education

Bachelor's degree