Description

You will design, implement, and manage data architectures and CI/CD pipelines within the Azure ecosystem.

Responsibilities

• Build and maintain CI/CD pipelines for Databricks and Azure Data Factory using Jenkins, Bitbucket, and Azure DevOps.
• Develop and monitor ETL pipelines in Azure Data Factory, including copy, lookup, validation, and switch activities.
• Manage Databricks assets by implementing PySpark, SQL, and Python notebooks.
• Implement security protocols using Azure Key Vault, secrets, and scopes to meet business requirements.
• Troubleshoot pipeline performance and resolve issues related to Azure deployments and scalability.

Required Skills

• 10+ years of experience in data engineering or architecture.
• Proficiency in PySpark, SQL, and Python.
• Hands-on experience with Azure Data Factory (ETL, triggers, datalinks) and Azure Databricks (Clusters, Notebooks, Workspaces, Delta Lake).
• Expertise in CI/CD implementation using Bitbucket, Jenkins, and Azure DevOps.
• Strong knowledge of Azure storage services, virtual networks, and identity and access management (IAM).
• Experience with Databricks CLI, REST APIs, and automation via Python or Azure CLI.
• Understanding of Azure governance, policies, and cost control frameworks.

Preferred Skills

• Certifications in Azure, Databricks, or DevOps.
• Experience with GitHub Actions.

Key Skills

Education

Any Graduate