Description

You will design and implement end-to-end large-scale data systems and scalable processing pipelines.

This role is on-site.

Responsibilities

• Implement end-to-end large-scale data projects and system architectures.
• Develop scalable, high-performance code for data processing.
• Execute parallel batch processing using Python.
• Manage data workflows using AWS Batch and Step Functions.

Required Skills

• 5+ years of experience in data engineering.
• Expertise in Python programming and PySpark.
• Proficiency with Pandas, NumPy, and Joblib libraries.
• Hands-on experience with AWS EMR and PySpark.
• Experience with AWS Batch and AWS Step Functions.
• Proven track record of implementing 2 or more large-scale projects.
• Strong analytical and problem-solving skills.

Preferred Skills

• Experience with Apache Airflow.

Key Skills

Education

Any Graduate