Description

• We are looking for an experienced Azure Databricks Platform Engineer to design, implement, administer, automate, and support enterprise-scale Azure Databricks environments.
• The engineer will work closely with data engineering, cloud infrastructure, security, networking, DevOps, and application teams to provide a secure, scalable, governed, reliable, and cost-efficient Databricks platform.
• Design, configure, administer, and support Azure Databricks workspaces across development, testing/UAT, and production environments.
• Manage Databricks compute, cluster policies, jobs, workflows, user access, monitoring, and cost controls.

• Implement and administer Unity Catalog, including catalogs, schemas, permissions, storage credentials, and external locations.
• Integrate Azure Databricks with ADLS Gen2, Azure Data Factory, Azure SQL, Azure Key Vault, and managed identities. Implement secure authentication and authorization using Microsoft Entra ID, Azure RBAC, managed identities, and Key Vault.

• Work with Azure networking concepts including VNets, NSGs, private endpoints, DNS, routing, VPN/ExpressRoute, and hub-and-spoke connectivity.
• Build automation using Python, PowerShell, Bash, Terraform/Bicep and CI/CD tooling to reduce manual platform activities.
• Infrastructure-as-code and scripting are also highlighted in the related internal Databricks requirement.
• Support CI/CD and release management using Azure DevOps, GitHub Actions, or equivalent tooling.
• Establish platform monitoring and observability using Azure Monitor, Log Analytics, Application Insights, Grafana, Prometheus, Datadog, Dynatrace, or New Relic as applicable.
• Monitor platform availability, performance, capacity, and operational health.
• Troubleshoot complex Databricks, Azure infrastructure, storage, identity, networking, and connectivity issues. Implement platform-level security, governance, access controls, and operational standards

Key Skills

Education

Bachelor's degree