Description

Key Responsibilities
Platform & Infrastructure
Design, deploy, and manage Databricks workspaces across multiple Azure subscriptions and environments (dev, staging, production)
Implement Infrastructure-as-Code (IaC) solutions using Terraform for Databricks resources
Establish and maintain security best practices, including network isolation, RBAC, and data governance
Configure and optimize cluster management, including auto-scaling policies and cost optimization strategies
Manage Databricks workspace connectivity and networking (VNets, private endpoints, firewalls)

DevOps & Automation
Build CI/CD pipelines using Azure DevOps for deploying Databricks notebooks, jobs, and configurations
Implement automated testing and validation of data pipelines
Create self-service deployment frameworks and templates for development teams
Establish monitoring, alerting, and logging solutions for Databricks environments
Automate workspace provisioning and deprovisioning processes

Collaboration & Enablement
Partner with data engineers and analysts to understand platform requirements
Provide technical guidance and documentation for platform usage
Develop runbooks and operational procedures for incident response
Mentor team members on Databricks best practices and Azure DevOps tools

Governance & Compliance
Implement access controls and audit logging across Databricks workspaces
Ensure compliance with organizational security and data governance policies
Monitor and manage workspace usage, costs, and resource utilization
Document architecture decisions and maintain platform documentation

Required Qualifications
7+ years of experience in platform engineering, infrastructure engineering, or DevOps
2+ years of hands-on experience with Databricks infrastructure setup (or Apache Spark/Delta Lake environments)
3+ years of experience with Azure cloud services and infrastructure
2+ years of experience with Azure DevOps (Pipelines, Repos, Artifacts)
Strong proficiency in Infrastructure-as-Code tools (Terraform Enterprise)
Solid understanding of CI/CD principles and implementation
Experience with Linux/Unix environments and scripting (Bash, Python, or similar)
Familiarity with containerization technologies (Docker, Kubernetes)
Strong knowledge of networking, security, and identity management in cloud environments

Preferred Qualifications
Experience with Unity Catalog and Databricks governance features
Knowledge of Apache Spark architecture and optimization
Experience with other IaC tools (Terraform)
Familiarity with cost optimization strategies in cloud environments
Azure certifications (AZ-900, AZ-104, AZ-305)
Databricks Certification
Experience with Git and version control workflows
Knowledge of data governance and data mesh architectures
Experience with monitoring tools (Datadog, New Relic, Azure Monitor)
 

Key Skills

Education

Bachelor's degree