Description

You will own the migration of legacy ETL workflows from Ab Initio to PySpark-based pipelines on Databricks.

Responsibilities

• Refactor monolithic processes into modular, reusable components.
• Build and maintain ETL/ELT pipelines integrating data from Snowflake and other sources.
• Create end-to-end data lineage and flow diagrams to ensure governance and quality standards.
• Develop unit, integration, and reconciliation frameworks for validation.
• Support cutover strategies and decommission legacy workflows.

Required Skills

• 5+ years of experience in data engineering.
• Strong proficiency in PySpark for scalable data pipeline development.
• Experience with Snowflake as a data source.
• Knowledge of data governance principles.
• Ability to collaborate effectively with data architects and analysts.
• Bachelor's degree in a relevant field.

Key Skills

Education

Bachelor's degree