Description

You will design, develop, and maintain data pipelines for integration and ingestion across various sources. You will implement data cleaning, enrichment, and aggregation processes using Bronze, Silver, and Gold architecture layers. You will manage data within Dimension and Fact tables, ensuring integrity and updating Entity-Relationship models. You will migrate local data platforms to a Common Data Platform (CDP) and replicate existing ETL transformations. You will collaborate with Technical and Business QA teams to validate pipelines and troubleshoot data-related issues.

Responsibilities

• Design and maintain data pipelines for integration and ingestion.
• Implement data cleaning and aggregation using Bronze, Silver, and Gold layers.
• Manage Dimension and Fact tables and update Entity-Relationship models.
• Migrate local data platforms to a Common Data Platform (CDP).
• Collaborate with QA teams to validate pipelines and troubleshoot issues.

Required Skills

• Proficiency in PySpark and SQL.
• Experience with Python and Bash scripting.
• Hands-on knowledge of AWS.
• Experience with CI/CD using GitHub Actions.
• Understanding of Medallion Architecture for data structuring.
• Knowledge of graph databases such as Neptune or Neo4j.
• Experience with Atlassian or Unity Catalog for data governance.
• Bachelor's degree in Computer Science, Information Technology, or a related field.
• 12+ years of experience.

Preferred Skills

• Experience with Talend, Informatica, or similar ETL tools.
• Familiarity with Terraform for infrastructure management and Airflow for workflow automation.

Key Skills

Education

Bachelor's Degree