Description

You will design and implement distributed data processing pipelines and data products to handle large volumes.

Responsibilities

• Design and implement distributed data processing pipelines and data products in collaboration with product owners and analysts.
• Contribute to the architecture, frameworks, and patterns used for processing and storing large datasets.
• Translate product backlog items into engineering designs and logical units of work.
• Build utilities, user-defined functions, libraries, and frameworks to support data flow patterns.
• Implement automated routines using workflow orchestration tools and incorporate automated unit testing.

Required Skills

• 5+ years of experience in big data engineering.
• Proficiency in Python.
• Strong SQL skills.
• Experience with Spark.
• Ability to profile and analyze data to design scalable solutions.
• Experience defining data acquisition and consumption strategies.
• Bachelor's degree.

Key Skills

Education

Bachelor's degree