Description

You will design, implement, and manage scalable and secure Databricks platforms to ensure optimal performance and reliability.

Responsibilities

• Create and manage Databricks workspaces and clusters, handling resource right-sizing and lifecycle management.
• Automate infrastructure provisioning and deployment using Infrastructure-as-Code tools.
• Monitor platform health, performance, and cost by building dashboards and integrating monitoring tools.
• Troubleshoot and resolve issues related to cluster configuration, job failures, and performance bottlenecks.
• Implement security and governance policies, including user management and access controls using Unity Catalog.

Required Skills

• 5+ years of hands-on experience architecting, deploying, managing, and supporting Azure Databricks Platform and Azure Data Lake.
• Required experience working with Azure Cloud provider tools and architecture.
• Hands-on experience using scripting languages and CLI tools including Terraform, Python, Bash, PowerShell, Ansible, SQL, Azure CLI, and Databricks CLI.
• Experience with implementing CI/CD pipelines using GitLab CI/CD.
• Knowledge of Delta Lake and Photon for resource optimization.
• Familiarity with Agile product development team environments.
• Ability to articulate system designs and patterns to various leadership levels.

Key Skills

Education

Any Graduate