Description

Lead the modernization of legacy Cloudera data platforms to Databricks Lakehouse architecture, owning end-to-end data engineering strategy and execution.

This role is on-site.

Responsibilities

• Redesign ingestion, transformation, and consumption patterns from HDFS-based architectures to cloud object storage and Delta Lake.
• Refactor legacy Hive and Impala logic into PySpark and Spark SQL ELT pipelines while ensuring data reconciliation and audit integrity.
• Design and govern enterprise Data Warehouse and Lakehouse architectures, implementing layered models (Raw, Curated, Semantic).
• Optimize large-scale data pipelines using Z-ORDER, OPTIMIZE, and cluster configurations to ensure performance and cost-efficiency.
• Establish data governance, lineage, metadata management, and security standards across the platform.

Required Skills

• 12+ years of overall data engineering experience.
• 8+ years of experience with enterprise Data Warehouse and Data Lake platforms.
• 5+ years of hands-on experience with Databricks and Spark at scale.
• Strong experience modernizing legacy Cloudera platforms (CDH/CDP, Hive, HBase, Impala) to Databricks.
• Proficiency in PySpark, Spark SQL, and Delta Lake.
• Experience with cloud platforms such as AWS or Azure.
• Knowledge of CI/CD pipelines using Git, Terraform, Jenkins, or Azure DevOps.
• Familiarity with orchestration tools like Airflow or Databricks Workflows.
• Bachelor's degree in a relevant field.

Preferred Skills

• Experience with finance and risk data models (General Ledger, Sub-ledger, credit/liquidity/market risk).
• Experience with dbt or similar frameworks for semantic layer development.

Key Skills

Education

Bachelor's degree