Description

You will own the end-to-end architecture and implementation of enterprise-scale Databricks platforms, driving scalability, governance, and cost optimization across data ecosystems.

Responsibilities

• Architect and implement secure, scalable Lakehouse environments across dev, test, and prod using Delta Lake and Unity Catalog.
• Design workspace strategies, cluster policies, and job orchestration frameworks to support high-performance data engineering.
• Build and optimize batch and real-time streaming pipelines using Apache Spark (PySpark/Scala) and Structured Streaming.
• Implement CI/CD pipelines for Databricks using Azure DevOps, Jenkins, or GitHub Actions and enable Infrastructure as Code.
• Optimize workloads for performance and cost using partitioning, caching, Photon engine, and efficient cluster strategies.

Required Skills

• 10+ years of experience in data engineering or architecture roles.
• Advanced proficiency in SQL and Python.
• Hands-on experience with Databricks, Delta Lake, Medallion Architecture, and Unity Catalog.
• Strong experience with at least one major cloud platform: AWS, Azure, or GCP.
• Experience with CI/CD tools including Azure DevOps, Jenkins, or GitHub Actions.
• Knowledge of workflow orchestration tools such as Airflow.
• Experience with Infrastructure as Code (Terraform or equivalent).
• Ability to design secure, scalable Lakehouse architectures integrating with cloud-native services.

Preferred Skills

• Databricks certifications (Associate or Professional).
• Experience with Databricks SQL, dashboards, or MLflow.
• Domain experience in Finance, Retail, or Healthcare.

Key Skills

Education

Any Graduate