Description

You will build and maintain data pipelines using PySpark and Data Vault 2.0 methodologies.

Responsibilities

• Build ETL pipelines to move data from source systems into AWS S3 staging buckets.
• Develop data ingestion frameworks into raw tables using advanced PySpark programs.
• Design ETL pipelines using PySpark, Scala, and Snowsql to transform data from Hubs and Satellites into Data Vault 2.0 structures.
• Implement data quality checks, SOX controls, and batch job scheduling via Airflow and Autosys.
• Design test cases and execute testing techniques to validate all development.

Required Skills

• 5+ years of experience in data engineering.
• Proficiency with PySpark for data ingestion and transformation.
• Experience with Scala and Snowsql.
• Hands-on work with AWS S3.
• Direct experience building ETL pipelines.
• Working knowledge of Airflow and Autosys for job scheduling.
• Ability to work with Data Vault 2.0 modeling, including Hubs and Satellites.
• Bachelor’s degree in Computer Science, Computer Information Systems, or a related engineering field.

Key Skills

Education

Any Graduate