Description

You will build and maintain data pipelines using the AWS ecosystem and PySpark.

Responsibilities

• Develop and manage Airflow tasks for data orchestration.
• Write and optimize PySpark jobs for large-scale data processing.
• Build infrastructure using CloudFormation and manage containerized workloads via EKS and ECR.
• Perform general scripting and data manipulation using Python libraries.

Required Skills

• 10+ years of professional experience in data engineering.
• Expertise in PySpark and AWS EMR.
• Strong proficiency in SQL.
• Deep knowledge of Python and libraries including Pandas, NumPy, and Boto.
• Hands-on experience with AWS S3, RDS, Athena, and Lambda.
• Experience with AWS CloudFormation and EKS.
• Degree in any field of study.

Key Skills

Education

Any Graduate