Description

Key Responsibilities

Architectural Strategy & Governance

• Design the target Databricks Lakehouse architecture utilizing Delta Lake, Photon, and Unity Catalog.
• Establish global code refactoring standards, optimization benchmarks, and PySpark best practices.
• Resolve highly complex dependency mappings and architect seamless, zero-downtime dual-run strategies.
• Lead the technical deployment and integration of specialized migration accelerators

Hands-on Engineering & Optimization

• Review automated output from migration tools and manually refactor complex legacy logic into high-performing PySpark notebooks.
• Eliminate legacy anti-patterns such as massive row-by-row processing and inefficient lookups.
• Optimize PySpark code performance using advanced Spark features including Z-Ordering, partitioning, and caching.
• Build robust Databricks Workflows and orchestrate complex DAGs based on comprehensive source lineage.

Technical Skills & Competencies

• Core Platforms: Databricks, Delta Lake, Unity Catalog, Photon, DataStage.
• Languages & Frameworks: PySpark, Python, SQL, Shell Scripting.
• Cloud & DevOps: AWS alongside CI/CD deployment pipelines.
• Orchestration: Apache Airflow, Databricks Workflows

Key Skills

Education

Bachelor's degree