Description

You will build and maintain scalable data pipelines and processing workflows within an AWS environment.

Responsibilities

• Design and develop scalable data pipelines using Apache Spark and ETL tools.
• Implement data processing workflows with Apache Airflow to manage ingestion and transformation.
• Write efficient transformation scripts in Scala and Python to process large datasets.
• Manage data storage and services using AWS, including S3 and Redshift.
• Develop CI/CD pipelines to automate data deployment and monitor pipeline performance.

Required Skills

• 5+ years of experience in Data Engineering.
• Proficiency with Apache Spark and ETL processes.
• Hands-on experience with Apache Airflow for workflow orchestration.
• Strong programming skills in Python and Scala.
• Experience managing AWS services, specifically S3 and Redshift.
• Ability to build and maintain CI/CD pipelines.
• Bachelor's degree.

Key Skills

Education

Bachelor's degree