Description

Job Description

• Minimum 4+ years of hands-on experience administering Azure Databricks platforms in an enterprise environment.
• Proven experience configuring and managing Azure Databricks platform components, including:
• Clusters: autoscaling configurations, instance pools, cluster policies, and performance optimization
• Jobs and Workflows: scheduling, concurrency controls, retries, alerts, and notifications
• Workspace assets: notebooks, Git repos, libraries, init scripts, and secret scopes
• Strong experience with Unity Catalog and implementation of enterprise data governance models, including fine grained access controls and catalog level security.

Experience with Databricks automation and deployment tooling, including:

• Databricks CLI
• Databricks REST APIs
• Databricks Asset Bundles (DAB)
• Experience integrating Databricks with BI Tools/analytics platforms.
• Solid understanding of data engineering patterns, Spark based workloads, and techniques for tuning performance and optimizing resource usage.

Working knowledge of Microsoft Azure fundamentals, including:

• Identity and access management using RBAC, service principals, and managed identities
• Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Key Vault, Azure Monitor, and Log Analytics
• Azure networking concepts such as VNets, private endpoints, and DNS resolution
• Proven expertise with Terraform, including:
• Modular design patterns
• Remote state management
• Multi environment deployment strategies
• Secure infrastructure as code practices

Strong experience building and maintaining CI/CD pipelines using GitLab CI/CD, including:

• YAML based pipeline definitions
• Runner configuration
• Environment promotion strategies
• Approval gates and controlled deployments

Demonstrated experience supporting compliance driven and regulated environments, including:

• Audit evidence collection and documentation
• Periodic access reviews
• Change management and release controls

Nice to Have experience on MLOps, including:

• Experience supporting machine learning workloads on Azure Databricks
• Familiarity with MLflow for experiment tracking, model registry, and lifecycle management
• Exposure to MLOps pipelines, including model training, validation, and deployment automation
• Understanding of model governance, versioning, and promotion across environments

Key Skills

Education

Any Graduate