Description

You will engineer and maintain the Databricks platform infrastructure and deployment pipelines.

Responsibilities

• Design and standardize Dev/UAT/Prod Databricks workspaces, including networking, Private Link, and Key Vault integrations.
• Implement YAML-based Azure DevOps pipelines to automate notebook and job deployments, environment promotions, and compliance checks.
• Author reusable Terraform or Bicep modules to manage workspaces, clusters, and Unity Catalog objects.
• Establish monitoring, alerting, and automation for cluster SLAs, autoscaling, and disaster recovery scenarios.
• Configure Unity Catalog governance, including catalogs, schemas, RBAC, and data access patterns.

Required Skills

• 6+ years of experience in a DevOps or Platform Engineering role focused on Azure.
• Deep expertise with Azure Databricks, Azure Data Lake Storage, and Azure Resource Manager (ARM).
• Strong proficiency in CI/CD practices using Azure DevOps and YAML pipelines.
• Hands-on experience automating Databricks artifacts (clusters, jobs, notebooks) using pipelines.
• Expertise in Infrastructure as Code using Terraform, ARM templates, or Bicep.
• Proficiency in scripting using Python and PowerShell.
• Familiarity with Apache Spark (PySpark, Spark SQL).
• Experience implementing cloud security best practices: RBAC, managed identities, and Key Vault.
• Solid understanding of Git branching strategies and conflict resolution.

Key Skills

Education

Any Graduate