Description

Key Skills: AWS, Pyspark

Roles and Responsibilities:

• Design, develop, and maintain scalable ETL pipelines using PySpark for distributed data processing.
• Build and optimize big data workflows on AWS, leveraging services such as EMR and S3.
• Implement data ingestion and transformation patterns, including orchestration and job execution strategies.
• Integrate pipelines with AWS services for event-driven or scheduled processing (e.g., Lambda where applicable).
• Monitor pipeline performance, troubleshoot failures, and improve reliability and efficiency in production.

Skills Required:

• 6 - 9 years of relevant experience in AWS-based data engineering and ETL development
• AWS (core cloud services used for data processing and storage)
• Distributed processing on AWS using EMR and data storage/transfer with S3
• Performance optimization for large-scale ETL workloads
• Cloud-based pipeline execution and operational troubleshooting

Good to Have:

• PySpark for building Spark-based ETL and transformation jobs

Education: Degree in Computer Science, Engineering, or related field (or equivalent practical experience)

Key Skills

Education

Any Graduate