Description

You will architect and optimize Databricks Lakehouse solutions, leading migration efforts and ensuring high-performance data engineering practices.

This role is on-site.

Responsibilities

• Design target Databricks Lakehouse architecture using Delta Lake, Photon, and Unity Catalog.
• Establish global code refactoring standards and resolve complex dependency mappings for zero-downtime migrations.
• Refactor legacy logic into high-performing PySpark notebooks, eliminating inefficient anti-patterns.
• Optimize PySpark performance using Z-Ordering, partitioning, and caching strategies.
• Build Databricks Workflows and orchestrate complex DAGs using Apache Airflow.

Required Skills

• 5+ years of experience with Databricks, Delta Lake, and Unity Catalog.
• Strong proficiency in PySpark, Python, and SQL.
• Experience with AWS cloud infrastructure and CI/CD deployment pipelines.
• Expertise in Apache Airflow and Databricks Workflows for orchestration.
• Knowledge of Shell Scripting and data migration accelerators.
• Familiarity with DataStage and legacy system integration.
• Bachelor's degree in a relevant field.

Preferred Skills

• Experience with Photon engine optimization.
• Background in architectural strategy and governance for large-scale data platforms.

Key Skills

Education

Bachelor's degree