Description

Lead the modernization of enterprise data platforms from legacy Cloudera architectures to Databricks Lakehouse, owning the end-to-end design of scalable data warehouses and lakehouses.

This role is on-site.

Responsibilities

• Redesign ingestion, transformation, and consumption patterns by migrating HDFS-based architectures to cloud object storage and Delta Lake.
• Refactor legacy Hive and Impala logic into PySpark and Spark SQL ELT pipelines, ensuring data reconciliation, audit integrity, and consistency.
• Design and govern layered lakehouse architectures (Raw/Landing, Curated/Conformed, Semantic/Consumption) and implement Medallion patterns.
• Optimize large-scale workloads using Z-ORDER, OPTIMIZE, caching, and cluster configurations while establishing data governance and lineage.
• Mentor senior engineers, define engineering standards, and act as the technical authority for architecture decisions.

Required Skills

• 12–18 years of data engineering experience, including 8+ years with enterprise Data Warehouse and Data Lake platforms.
• 5+ years of hands-on experience with Databricks and Spark at scale.
• Strong expertise in modernizing legacy Cloudera platforms (CDH/CDP, Hive, HBase, Impala) to Databricks.
• Deep knowledge of finance and risk data models, including General Ledger, sub-ledgers, and risk exposure models.
• Experience implementing semantic layers for BI, reporting, and analytics with defined business metrics and KPIs.
• Proficiency with cloud platforms (AWS or Azure) and CI/CD pipelines using Git, Terraform, Jenkins, or Azure DevOps.
• Experience with orchestration tools such as Airflow or Databricks Workflows.

Preferred Skills

• Experience with dbt for transformation frameworks.
• Background in BFSI, Capital Markets, or regulatory reporting.

Key Skills

Education

Bachelor's degree