Description

You will design, implement, and maintain big data infrastructure and processing pipelines scaling to billions of structured and unstructured records.

Responsibilities

• Build data pipelines to automate ingestion, transformation, and augmentation of data sources.
• Implement best practices for pipeline operations and system maintenance.
• Oversee assigned programs and provide technical guidance to team members.
• Solve complex technical problems and explore new technologies to address business requirements.
• Apply data modeling, data mining, and segmentation techniques to large datasets.

Required Skills

• 5+ years of experience in Data Engineering.
• Hands-on experience with Spark and Scala.
• Proficiency in Python and Hive.
• Strong experience with Linux and Shell scripting.
• Experience designing high-performance big data systems.
• Degree in any graduate field.

Preferred Skills

• Experience with Spark Streaming or other live stream processing technologies.
• Experience building Flask and REST APIs.
• Cloud development experience with AWS, Azure, or GCP.

Key Skills

Education

Any graduate