Description

You will manage and optimize a cloud-based Databricks environment on Azure, ensuring platform reliability, security, and scalability.

Responsibilities

• Administer Databricks workspaces, including user management, access control, and security configurations.
• Manage and optimize Databricks clusters, focusing on configuration, scaling, and cost management.
• Maintain underlying Azure infrastructure such as virtual networks, storage accounts, and security groups.
• Manage integrations and data pipelines between Databricks and Snowflake.
• Develop automation scripts for platform provisioning and configuration using Terraform, Azure CLI, or Python.
• Monitor system health and implement alerting systems to identify performance bottlenecks.

Required Skills

• 5+ years of experience in platform administration or data engineering.
• Extensive experience managing Databricks workspaces on Azure.
• Proficiency with Snowflake data warehousing and integrations.
• Hands-on experience with Azure services including ADLS Gen2, Blob Storage, Azure AD, and Key Vault.
• Strong knowledge of Apache Spark.
• Experience with Infrastructure as Code (IaC) using Terraform or Azure CLI.
• Programming and scripting skills in Python, Bash, or Powershell.
• Solid understanding of networking fundamentals including TCP/IP, DNS, and Firewalls.

Key Skills

Education

Any Gradute