Description

You will own the design, implementation, and maintenance of large-scale data processing infrastructure.

Responsibilities

• Configure and optimize Databricks Pipelines for large-scale data processing within Azure Data Lakehouse.
• Set up and manage core Azure infrastructure components (Databricks Workspaces, AKS/ACI, Storage Accounts, Networking).
• Design and implement monitoring and observability frameworks using Azure Monitor, Log Analytics, and Prometheus/Grafana.
• Collaborate with engineering teams to enable microservices-based architecture for modular data solutions.
• Drive automation and CI/CD practices using Terraform, ARM templates, and GitHub Actions/Azure DevOps.

Required Skills

• 5+ years of hands-on experience with Azure Databricks, Delta Lake, and Apache Spark.
• Deep knowledge of Azure services, including Resource Manager, AKS, ACR, Key Vault, and Networking.
• Proven experience designing and implementing microservices architecture and container orchestration.
• Expertise in infrastructure-as-code, scripting (Python, Bash), and DevOps tooling (GitHub Actions, Azure DevOps).
• Familiarity with data governance, security, and cost optimization in cloud environments.
• Experience with Kafka is required.

Key Skills

Education

Any Graduate