Description

You will design and lead large-scale data architecture implementations using Databricks and modern data patterns.

This role is remote.

Responsibilities

• Architect modern data systems utilizing Lakehouse, Medallion, and Data Mesh patterns.
• Lead large-scale migrations involving over 100 pipelines from EDW platforms like Snowflake or Redshift.
• Optimize Spark performance and execute complex query tuning.
• Implement structured streaming and manage Slowly Changing Dimensions (SCD) types.
• Integrate data workflows with Git and CI/CD pipelines.

Required Skills

• 5+ years of experience in data engineering or architecture.
• Expertise in Databricks technologies including Unity Catalog, Delta Live Tables (DLT), Databricks SQL, and Auto Loader.
• Strong programming skills in Python; Scala is a significant plus.
• Deep knowledge of Spark tuning, Photon engine, and Delta Lake.
• Experience with data formats such as Iceberg, Parquet, and Delta.
• Proficiency in SQL and DBT.
• Hands-on experience with both Azure and AWS cloud environments.
• Proven experience in customer-facing consulting roles.
• Familiarity with Enterprise Data Warehousing (EDW) platforms.

Preferred Skills

• Experience with Terraform for infrastructure as code.

Key Skills

Education

Any Gradute