Description

You will administer and support Databricks platforms across Azure and AWS, owning security, governance, and performance optimization.

This role is remote.

Responsibilities

• Administer Databricks environments, implementing Unity Catalog, RBAC, and data governance controls.
• Design and optimize Spark and Delta Lake solutions for batch and real-time ingestion using Kafka and Spark Streaming.
• Develop Infrastructure as Code using Terraform, Ansible, and Python to manage provisioning, upgrades, and migrations.
• Manage cloud networking, including VNETs, private endpoints, DNS, and firewalls, while driving FinOps and cost optimization.
• Lead platform adoption, troubleshoot complex issues, and mentor junior engineers on technical standards.

Required Skills

• 10+ years of experience in data engineering or platform administration.
• Deep expertise in Spark, Delta Lake, and enterprise data processing.
• Proficiency with Kafka and Spark Streaming for real-time data pipelines.
• Strong command of Infrastructure as Code tools: Terraform and Ansible.
• Hands-on experience with Python for automation and scripting.
• Knowledge of cloud networking components: DNS, Firewalls, and VNETs.
• Experience with Azure and AWS Databricks deployments.

Preferred Skills

• Experience with Kubernetes and CI/CD integration for cloud-native deployments.
• Background in implementing data masking and audit readiness compliance.

Key Skills

Education

Any Graduate