Description

You will design, build, and maintain large-scale data pipelines on Azure Databricks, migrating legacy IBM DataStage workflows to modern PySpark and Python solutions.

Responsibilities

• Develop, test, and optimize ETL/ELT workloads in Azure Databricks, managing cluster configuration, autoscaling, and job scheduling.
• Migrate and validate legacy IBM DataStage code to Databricks, ensuring functional parity and performance improvements.
• Implement Infrastructure-as-Code using ARM templates and manage CI/CD pipelines via GitLab and Azure DevOps.
• Build secure data integration architectures using Azure services including Storage, Synapse, Key Vault, and Managed Identities.
• Configure monitoring, alerting, and cost optimization dashboards using Log Analytics and implement on-call support for production issues.

Required Skills

• 3+ years of experience building data pipelines with PySpark and Python in Azure Databricks.
• Advanced SQL proficiency for analytics and complex ETL processes.
• Hands-on experience with Azure core services: Storage Accounts, Synapse, Key Vault, VMSS, Function Apps, and Web Apps.
• Proficiency in Azure networking concepts, secure integration, and container services (ACA, container instances).
• Experience creating and deploying Azure infrastructure using ARM templates.
• Strong working knowledge of CI/CD workflows using GitLab and Azure DevOps.
• Awareness of IBM DataStage ETL/ELT tools and legacy code migration strategies.
• Experience implementing secure data access patterns with Azure Active Directory and service principals.

Preferred Skills

• Experience working in Agile teams with a focus on data governance and compliance frameworks.
• Strong analytical skills with proven ability to collaborate cross-functionally with stakeholders and legacy application teams.

Key Skills

Education

Not specified