Description

Execute large-volume data transformations and automation within an AWS environment, owning the data pipeline lifecycle from ingestion to storage.

This role is on-site.

Responsibilities

• Execute large-volume data transformations using Spark and Scala.
• Develop and manage AWS Glue jobs to run Spark applications.
• Build and orchestrate workflows using AWS Step Functions and Lambda.
• Automate testing processes using Python Behave and Scala.
• Manage data movement and storage through S3.

Required Skills

• 5+ years of experience in Data Engineering.
• Proficiency in Scala and the Spark framework.
• Strong Python skills for automation and Lambda functions.
• Hands-on experience with AWS Glue, Step Functions, Lambda, and Fargate/ECS.
• Expertise in SQL for data transformation and scripting.
• Experience with S3 for data management.
• Knowledge of Machine Learning Algorithms.

Key Skills

Education

Any Graduate