Description

Design, develop, and maintain scalable ETL pipelines using PySpark for distributed data processing on AWS.

This role is hybrid.

Responsibilities

• Build and optimize big data workflows on AWS, leveraging EMR and S3.
• Implement data ingestion and transformation patterns with orchestration and job execution strategies.
• Integrate pipelines with AWS services for event-driven or scheduled processing, including Lambda.
• Monitor pipeline performance, troubleshoot failures, and improve reliability in production.

Required Skills

• 6-9 years of experience in AWS-based data engineering and ETL development.
• Strong proficiency with PySpark for building Spark-based ETL and transformation jobs.
• Experience with distributed processing on AWS using EMR.
• Hands-on experience with S3 for data storage and transfer.
• Ability to perform performance optimization for large-scale ETL workloads.
• Skills in cloud-based pipeline execution and operational troubleshooting.
• Core knowledge of AWS services used for data processing and storage.

Preferred Skills

• Degree in Computer Science, Engineering, or related field (or equivalent practical experience).

Key Skills

Education

Any Graduate