Description

You will lead the migration of legacy ETL workflows from Ab Initio to PySpark-based pipelines on Databricks, ensuring data governance and quality standards are met.

This role is on-site.

Responsibilities

• Refactor monolithic Ab Initio processes into modular, reusable PySpark components.
• Build and maintain ETL/ELT pipelines integrating data from Snowflake and other sources.
• Create end-to-end data lineage and flow diagrams to ensure governance and quality.
• Develop unit, integration, and reconciliation frameworks for validation.
• Support cutover strategies and decommission legacy workflows.

Required Skills

• 5+ years of experience in data engineering.
• Strong proficiency in PySpark for scalable data pipeline development.
• Experience with Snowflake as a primary data source.
• Knowledge of data governance principles and implementation.
• Ability to collaborate effectively with data architects and analysts.
• Bachelor's degree in a relevant field.

Key Skills

Education

Bachelor's degree