Description

You will focus on building and maintaining CI/CD pipelines for migrating and managing Azure Databricks and Azure Data Factory ETL processes.

Responsibilities

• Build and maintain CI/CD pipelines for Azure Databricks and ADF using Bitbucket, Jenkins, and Azure DevOps.
• Create, monitor, and manage Azure Data Factory components including ETL pipelines, triggers, and data links.
• Implement and run Databricks jobs using PySpark, SQL, and Python notebooks.
• Manage Azure storage services, security scopes, and secret implementations.
• Mentor and train team members on DevOps setup and configuration.

Required Skills

• 5+ years of experience in DevOps or Data Engineering roles.
• Proficiency with Azure Data Factory (copy, lookup, validation, and switch activities).
• Hands-on experience with Azure Databricks (Clusters, Notebooks, Workspaces, and Jobs).
• Strong skills in PySpark, SQL, and Python.
• Experience with Bitbucket and Jenkins for repository management and CI/CD.
• Knowledge of Azure security principles, IAM, and network security groups.
• Proficiency in scripting with PowerShell, Python, or Azure CLI.
• Understanding of Databricks Delta, CLI, and REST API for automation.
• Ability to design cloud architectures including virtual networks and compute in Azure.

Preferred Skills

• Certifications in Azure, Databricks, or DevOps.
• Direct experience with Azure Synapse integration.

Key Skills

Education

Any Graduate