Description

What you'll do:

• Architect, design, and implement scalable and performant Databricks-based data pipelines, including batch and streaming workloads.
• Optimize Databricks Lakehouse architecture — including Delta Lake, Unity Catalog, and Photon — for cost efficiency, data reliability, and query performance at scale.
• Define and implement advanced data models and ontologies to support complex analytics and AI/ML workflows.
• Design and tune complex Spark applications (PySpark and/or Scala) for high-volume data processing, including structured streaming and incremental processing.
• Implement orchestration workflows (e.g., Databricks Workflows, DLT, or similar) to automate data pipelines with robust monitoring and alerting.
• Drive best practices for data governance, security, and lineage, leveraging Unity Catalog, table ACLs, and audit capabilities.
• Optimize cluster configurations, autoscaling strategies, and cost management within Databricks environments.
• Collaborate with data scientists and analytics engineers to enable self-service access to high-quality, well-documented datasets and features.
• Lead complex troubleshooting efforts, deep performance tuning, and root-cause analysis across distributed systems.
• Author and maintain technical artifacts, including architecture diagrams, design docs, reference implementations, and best practice guides.
• Mentor and coach engineers across teams on Databricks internals, Spark performance tuning, and modern data engineering architectures.
• Stay on top of emerging Databricks features (e.g., Mosaic AI, serverless compute, Delta Live Tables), advocating for their adoption when they bring tangible value.

You'll be expected to have:

• Bachelor’s or higher degree in Computer Science, Software Engineering, or a related field.
• 7+ years of software engineering experience, with 5+ years specifically in building large-scale data platforms.
• Expert-level proficiency in Databricks, including:
• Delta Lake internals (ACID transactions, time travel, Z-ordering, data compaction strategies)
• Unity Catalog for data governance, security, and lineage
• Advanced Spark tuning (shuffle management, partitioning strategies, caching, cluster configuration)
• Databricks SQL Warehousing, Workflows, and MLflow integration
• Strong expertise with cloud-native data engineering in Azure or AWS, including storage (ADLS/S3), networking, and IAM integration.
• Knowledge of streaming architectures (Apache Kafka, structured streaming), schema evolution, and event-driven data modeling.
• Hands-on experience with metadata catalogs, orchestration frameworks, and observability tooling (e.g., Datadog, Grafana).
• Strong understanding of distributed systems, fault tolerance, and large-scale performance engineering.
• Proven ability to solve complex, ambiguous problems and make architectural trade-offs in real-world production systems.
• Excellent technical communication skills — ability to document, present, and explain technical concepts clearly to varied audiences.
• Prior experience mentoring engineers, conducting design reviews, and setting technical direction across teams.
• A continuous learner mindset — actively keeping pace with evolving Databricks platform capabilities and broader data engineering trends.

Preferred but not required:

• Experience with Power BI Direct Lake mode and semantic modelling for Lakehouse-based analytics.
• Familiarity with Mosaic AI, Databricks GenAI capabilities, or Feature Store implementations.
• Contributions to open-source Spark, Databricks community content, or certifications (e.g., Databricks Certified Data Engineer Professional)

Key Skills

Education

Any Graduate