Description

You will design, implement, and manage scalable and secure Databricks platforms to ensure optimal performance and reliability.

Responsibilities

• Create and manage Databricks workspaces and clusters, including resource right-sizing and lifecycle management.
• Automate infrastructure provisioning and deployment using Infrastructure-as-Code tools.
• Monitor platform health, performance, and cost by building dashboards and integrating with monitoring tools.
• Troubleshoot and resolve issues related to cluster configuration, job failures, and performance bottlenecks.
• Implement security and governance policies, including user management and access controls using Unity Catalog.

Required Skills

• 10+ years of hands-on experience architecting, deploying, and managing Azure Databricks Platform.
• Required hands-on experience with Azure Cloud provider tools and architecture.
• Proficiency in scripting languages and CLI tools: Python, Bash, PowerShell, Terraform, Azure CLI, Databricks CLI, and SQL.
• Experience with CI/CD Pipelines and Gitlab CI/CD.
• Knowledge of data ingestion and data pipeline patterns using tools like Kafka, Snowflake Services, and Databricks.
• Familiarity with Delta Lake and Photon for resource optimization.
• Experience working in an agile product development team.
• Understanding of Azure Datalake and related solutions.

Key Skills

Education

Any Graduate