Description

You design and implement distributed data processing pipelines and data products to handle large volumes.

None of the work mode flags (remote, hybrid, on-site) are set to 1, so no work mode line is included.

Responsibilities

• Design and implement distributed data processing pipelines and data products in collaboration with product owners and analysts.
• Contribute to the architecture, frameworks, and patterns used for processing and storing large datasets.
• Translate product backlog items into engineering designs and logical units of work.
• Build utilities, user-defined functions, libraries, and frameworks to support data flow patterns.
• Implement automated routines using workflow orchestration tools and incorporate automated unit testing.

Required Skills

• 5+ years of experience in big data engineering.
• Proficiency in Python.
• Strong SQL skills.
• Experience with Spark.
• Ability to profile and analyze data to design scalable solutions.
• Experience defining data acquisition and consumption strategies.
• Bachelor's degree.

Key Skills

Education

Bachelor's degree