Description

You will own the design, build, and maintenance of CI/CD pipelines for Azure Data Factory and Databricks notebooks.

Responsibilities

• Automate unit, integration, and QA testing workflows within Databricks.
• Implement and manage CI/CD pipelines specifically for schema updates, applying data architecture principles.
• Develop and maintain Azure DevOps Pipelines using YAML builds and release workflows.
• Write and maintain automation scripts using Python, PowerShell, and Terraform for infrastructure management.
• Monitor, troubleshoot, and resolve pipeline failures and deployment inconsistencies across environments.

Required Skills

• 12+ years of professional experience.
• Expertise in Azure DevOps for pipeline development.
• Strong proficiency in Python and PowerShell for scripting.
• Experience using Terraform for infrastructure-as-code implementation.
• Hands-on experience with Databricks notebooks and Azure Data Factory (ADF).
• Familiarity with Azure services including ADLS Gen2, Key Vault, and DABs.
• Ability to apply IaC best practices to provision cloud resources.

Key Skills

Education

Any Graduate