Description

You will build and maintain CI/CD pipelines for Azure Databricks and Azure Data Factory migrations, ensuring automated, reliable ETL deployments.

Responsibilities

• Design and implement CI/CD pipelines for Azure Data Factory (ADF) and Databricks using Azure DevOps, Bitbucket, and Jenkins.
• Create, monitor, and troubleshoot ADF components including copy, lookup, validation, and switch activities.
• Manage Azure Databricks assets by executing PySpark, SQL, and Python notebooks, and configuring clusters, workspaces, and jobs.
• Implement Azure security controls, including Key Vault, scopes, and network security groups, to meet business compliance requirements.
• Automate ETL deployment processes and monitor pipeline performance to ensure reliability and optimal execution.

Required Skills

• 6+ years of experience in Azure cloud development and DevOps engineering.
• Proficiency in Python, SQL, and PySpark for data engineering tasks.
• Hands-on experience with Azure Data Factory (ADF) and Azure Databricks (Delta Lake, REST API, CLI).
• Strong expertise in CI/CD tools: Azure DevOps, Jenkins, and Bitbucket (Git version control).
• Knowledge of Azure storage services, virtual networks, and identity/access management (IAM).
• Ability to script automation using PowerShell, Azure CLI, or Python.

Preferred Skills

• Experience mentoring teams on DevOps setup and configuration.
• Azure, Databricks, or DevOps certifications.

Key Skills

Education

Any Graduate