Description

You will build and manage data pipelines to process and move data across various systems.

Responsibilities

• Design, document, and integrate production and post-production tools with various departments.
• Develop event-driven data pipelines that trigger, validate, and move data to different S3 buckets.
• Standardize the vendor on-boarding process to streamline data ingestion using approved cloud tools.
• Develop and automate data validations across landing, raw, archive, and curated data layers in AWS.
• Maintain the Data Quality Framework, checking data quality rules and alerting stakeholders via email reports.

Required Skills

• Experience with AWS Glue and Redshift.
• Proficiency in PySpark, Python, and SQL.
• Working knowledge of AWS S3, SNS, and SQS.
• Familiarity with workflow tools like Jira and Code Pipeline.
• Experience developing ETL/ELT processes.
• Ability to manage data validation frameworks.
• Bachelor's or Master's degree in a related technical field.

Key Skills

Education

Any Graduate