Description

Architect and optimize scalable Databricks Lakehouse data pipelines and analytics platforms.

This role is hybrid.

Responsibilities

• Design and implement high-performance batch and streaming data pipelines using PySpark and Scala.
• Optimize Delta Lake internals, Unity Catalog governance, and Photon engine for cost and query performance.
• Tune complex Spark applications, managing shuffle, partitioning, and cluster autoscaling strategies.
• Define data models and ontologies to support advanced analytics and AI/ML workflows.
• Implement orchestration via Databricks Workflows and DLT with robust monitoring using Datadog or Grafana.

Required Skills

• 7+ years software engineering experience, including 5+ years building large-scale data platforms.
• Expert proficiency in Databricks, Delta Lake, Unity Catalog, and Spark SQL.
• Strong experience with Azure or AWS cloud-native data engineering (ADLS/S3, IAM, networking).
• Deep understanding of distributed systems, fault tolerance, and performance tuning.
• Hands-on experience with Apache Kafka and structured streaming architectures.
• Bachelor’s degree in Computer Science or related field.

Preferred Skills

• Experience with Power BI Direct Lake mode and semantic modelling.
• Familiarity with Mosaic AI, GenAI capabilities, or Feature Store implementations.

Key Skills

Education

Any Graduate