Description

Administer and optimize Azure Databricks platforms for enterprise data engineering and governance.

This role is on-site.

Responsibilities

• Configure and manage Azure Databricks clusters, including autoscaling, instance pools, policies, and performance tuning.
• Implement Unity Catalog for fine-grained access controls, catalog-level security, and enterprise data governance.
• Build and maintain CI/CD pipelines using GitLab CI/CD, including YAML definitions, runner configuration, and environment promotion.
• Manage workspace assets such as notebooks, Git repos, libraries, init scripts, and secret scopes.
• Deploy infrastructure using Terraform with modular design, remote state management, and secure IaC practices.

Required Skills

• 4+ years administering Azure Databricks in enterprise environments.
• Strong expertise in Unity Catalog and data governance models.
• Proficiency with Databricks CLI, REST APIs, and Databricks Asset Bundles (DAB).
• Experience with Terraform for multi-environment deployment strategies.
• Solid understanding of Spark-based workloads and performance optimization.
• Working knowledge of Azure fundamentals: RBAC, Service Principals, ADLS Gen2, Key Vault, Monitor, and Log Analytics.
• Familiarity with Azure networking (VNets, private endpoints, DNS).

Preferred Skills

• Experience with MLOps, MLflow, and machine learning workload support on Azure Databricks.
• Knowledge of compliance-driven environments, including audit evidence collection and access reviews.

Key Skills

Education

Any Graduate