Description

You will design, develop, and maintain end-to-end data pipelines and ETL/ELT workflows using PySpark and Python. You will own the migration of legacy DataStage code to Databricks, ensuring functional parity and performance.

This role is on-site.

Responsibilities

• Implement and optimize large-scale data processing workloads in Azure Databricks, managing cluster configuration, autoscaling, and governance.
• Build data integration and orchestration solutions using Azure services to meet performance, availability, and security requirements.
• Author Infrastructure-as-Code using ARM templates and Terraform to deploy consistent, repeatable Azure environments.
• Configure Azure components including Storage Accounts, Synapse, Key Vault, Function Apps, and Log Analytics for monitoring and cost optimization.
• Manage source control, CI/CD pipelines, and release management using GitLab and Azure DevOps while following Agile practices.

Required Skills

• 5+ years of experience with Python and PySpark for data engineering.
• Strong proficiency in building ETL/ELT workflows and SQL.
• Hands-on experience with Azure Databricks, Azure DevOps, and CI/CD pipelines.
• Experience with Infrastructure-as-Code tools such as ARM templates and Terraform.
• Familiarity with Azure networking, security (Active Directory, Managed Identities), and monitoring tools.
• Proficiency in Bash scripting and Agile/Scrum methodologies.
• Bachelor’s degree in any discipline.

Preferred Skills

• Experience with legacy DataStage migration and modernization.
• Knowledge of Azure Synapse and container services (ACI/Container Apps).
• Experience with production on-call support and incident resolution.

Key Skills

Education

Any Gradute