Description

You will define and own the architecture of data platforms built on Databricks, spanning ingestion, transformation, storage, and consumption layers.

Responsibilities

• Design and implement Lakehouse architectures using Delta Lake, Unity Catalog, and structured streaming.
• Architect scalable ETL/ELT pipelines utilizing Apache Spark, PySpark, Scala, and Databricks Workflows.
• Lead integration with enterprise systems, including data catalogs, ML platforms, BI tools, and cloud services.
• Guide teams on implementing CI/CD pipelines, version control, and automated testing for Databricks jobs.
• Provide thought leadership in performance tuning, cost optimization, and cluster configuration for Databricks environments.

Required Skills

• 5+ years of experience in data architecture.
• Deep expertise with Apache Spark and PySpark.
• Proficiency in Scala.
• Experience designing Lakehouse architectures (Delta Lake, Unity Catalog).
• Familiarity with cloud providers (Azure, AWS, GCP).
• Experience building and managing ETL/ELT pipelines.
• Knowledge of CI/CD practices for data engineering.
• Experience with ML lifecycle management tools.

Key Skills

Education

Any Gradute