Description

You will design and implement distributed data processing pipelines and own the end-to-end implementation of large-scale data solutions.

Responsibilities

• Design and implement distributed data processing pipelines using Spark, Hive, and Python within the Hadoop ecosystem.
• Develop and publish RESTful APIs using OpenAPI specifications to enable real-time data consumption across teams.
• Build proof of concepts using NoSQL technologies like HBase, DynamoDB, and Cassandra alongside stream processing frameworks.
• Contribute to DevOps by building utilities, user-defined functions, and frameworks to optimize data flow patterns.
• Ensure high-quality solutions through code reviews, documentation, and adherence to engineering best practices.

Required Skills

• 10+ years of experience in large-scale software development and Big Data technologies.
• Proficiency in Java, Scala, Python, Shell scripting, and SQL.
• Hands-on development experience with Spark, MapReduce, and Hive.
• Strong experience developing RESTful APIs.
• Experience with distributed stream processing frameworks such as Apache Spark, Flink, or Kafka Streams.
• Experience with NoSQL databases including HBase, DynamoDB, or Cassandra.
• Bachelor's degree in a relevant field.

Preferred Skills

• Experience with Business Rule management systems like Drools.

Key Skills

Education

Bachelor's degree