Description

You will build and manage data pipelines using Python and Spark on AWS.

Responsibilities

• Develop data processing jobs using PySpark and Python-based technologies.
• Implement data handling workflows using Pandas and Numpy.
• Manage data orchestration and workflows within the AWS ecosystem.
• Maintain and optimize big data projects using Spark as the core processing engine.

Required Skills

• 5+ years of professional experience in data engineering or related roles.
• Hands-on experience with PySpark and Spark core processing.
• Proficiency in Python and data libraries including Pandas and Numpy.
• Experience with AWS Glue and AWS Step Functions.
• Working knowledge of AWS Lambda, S3 (intelligent tiering), and EMR.
• Experience with AWS Lake Formation and Athena.
• Competency in Shell Scripting.
• Proven track record of delivering at least one big data project using Spark.

Preferred Skills

• Any Graduate degree.

Key Skills

Education

Any Graduate