Description

You will develop and maintain big data pipelines using PySpark and the Hadoop ecosystem.

Responsibilities

• Create, deploy, and maintain data pipelines using Spark and Hadoop.
• Handle large volumes of big data processing tasks.
• Schedule, monitor jobs, and build automation tools.
• Debug and resolve issues within the Spark and Hadoop environments.

Required Skills

• 5+ years of experience with PySpark and Spark within Hadoop distributed frameworks.
• 4+ years of experience using Scala and Python.
• Proficient knowledge of SQL with any RDBMS.
• Experience in building and maintaining data pipelines.
• Experience scheduling and monitoring data jobs.
• Familiarity with the Hadoop ecosystem.
• Ability to work effectively under deadlines and prioritize tasks.

Preferred Skills

• 1+ years of AWS development experience.
• Experience with DevOps or drive automation.

Key Skills

Education

Bachelor's degree