Description

Job Description

  • Minimum 4+ years of hands-on experience administering Azure Databricks platforms in an enterprise environment.
  • Proven experience configuring and managing Azure Databricks platform components, including:
  • Clusters: autoscaling configurations, instance pools, cluster policies, and performance optimization
  • Jobs and Workflows: scheduling, concurrency controls, retries, alerts, and notifications
  • Workspace assets: notebooks, Git repos, libraries, init scripts, and secret scopes
  • Strong experience with Unity Catalog and implementation of enterprise data governance models, including fine grained access controls and catalog level security.

Experience with Databricks automation and deployment tooling, including:

  • Databricks CLI
  • Databricks REST APIs
  • Databricks Asset Bundles (DAB)
  • Experience integrating Databricks with BI Tools/analytics platforms.
  • Solid understanding of data engineering patterns, Spark based workloads, and techniques for tuning performance and optimizing resource usage.

Working knowledge of Microsoft Azure fundamentals, including:

  • Identity and access management using RBAC, service principals, and managed identities
  • Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Key Vault, Azure Monitor, and Log Analytics
  • Azure networking concepts such as VNets, private endpoints, and DNS resolution
  • Proven expertise with Terraform, including:
  • Modular design patterns
  • Remote state management
  • Multi environment deployment strategies
  • Secure infrastructure as code practices

Strong experience building and maintaining CI/CD pipelines using GitLab CI/CD, including:

  • YAML based pipeline definitions
  • Runner configuration
  • Environment promotion strategies
  • Approval gates and controlled deployments

Demonstrated experience supporting compliance driven and regulated environments, including:

  • Audit evidence collection and documentation
  • Periodic access reviews
  • Change management and release controls


Nice to Have experience on MLOps, including:

  • Experience supporting machine learning workloads on Azure Databricks
  • Familiarity with MLflow for experiment tracking, model registry, and lifecycle management
  • Exposure to MLOps pipelines, including model training, validation, and deployment automation
  • Understanding of model governance, versioning, and promotion across environments

Key Skills
Education

Any Graduate

Back To Jobs