Description

You will design, build, and maintain data pipelines using AWS Glue and PySpark.

Responsibilities

• Design, build, and maintain data pipelines primarily using AWS Glue and PySpark.
• Source data from systems like Redshift and Aurora, performing streaming transformations and data cleaning.
• Deliver cleaned datasets by pushing them into S3 buckets.
• Manage secure file transfer via SFTP to external third-party servers, meeting deadlines.
• Consult with the team on efficient data output solutions within the AWS Glue/PySpark constraints.

Required Skills

• 5+ years of experience in Data Engineering.
• Heavy expertise in the AWS ecosystem, specifically AWS Glue.
• Hands-on experience implementing complex applications using PySpark.
• Strong knowledge of relational databases (Redshift, Aurora) and non-SQL databases.
• Demonstrated ability to efficiently ingest, transform, and output data.
• Experience managing data delivery via SFTP integrations.

Key Skills

Education

Any Gradute