Description

You will design and build end-to-end data platforms using Databricks Lakehouse architecture, Delta Lake, and Unity Catalog, supporting ML/AI workloads. You define ingestion, transformation, governance, and consumption frameworks aligned with enterprise standards. You evaluate architectural improvements, recommend new tools, and develop reference architectures and solution blueprints for Databricks implementations. You architect scalable ETL/ELT pipelines using PySpark, Spark SQL, Delta Live Tables, and Databricks Workflows. You implement performance optimization for notebooks, clusters, job orchestration, and data storage. You guide teams in building curated, governed data models using the Delta Lake Medallion Architecture.

Responsibilities

• Design end-to-end data platforms leveraging Databricks Lakehouse, Delta Lake, and Unity Catalog.
• Define data ingestion, transformation, governance, and consumption frameworks aligned to enterprise standards.
• Architect scalable ETL/ELT pipelines using PySpark, Spark SQL, Delta Live Tables, and Databricks Workflows.
• Implement performance optimization strategies for notebooks, clusters, job orchestration, and data storage.
• Guide teams in building curated, governed data models leveraging Delta Lake Medallion Architecture.

Required Skills

• 5+ years of experience in big data architecture and engineering.
• Strong proficiency in Databricks, including Delta Lake and Unity Catalog.
• Expertise in PySpark and Spark SQL for data processing.
• Experience designing and implementing ETL/ELT pipelines.
• Proficiency in SQL for data manipulation and querying.
• Bachelor's degree in Computer Science, Engineering, or related field.
• Ability to develop reference architectures and solution blueprints.

Key Skills

Education

Bachelor's degree