Description

Lead the design and implementation of end-to-end distributed data processing pipelines and architectures.

Responsibilities

• Design and implement distributed data processing pipelines using Spark, Hive, Python, and the Hadoop ecosystem.
• Build utilities, user-defined functions, and frameworks to optimize data flow patterns.
• Publish RESTful APIs using OpenAPI specifications to enable real-time data consumption.
• Collaborate with architecture and engineering leads to define and enforce engineering best practices.
• Develop end-to-end solutions incorporating distributed stream processing and NoSQL technologies.

Required Skills

• 8 to 10 years of experience in Big Data Management.
• Proficiency with AWS Services.
• Hands-on experience with Spark, Hive, and Python.
• Experience with NoSQL technologies such as HBase, DynamoDB, or Cassandra.
• Experience with distributed stream processing frameworks like Apache Spark, Flink, or Kafka.
• Experience publishing RESTful APIs via OpenAPI specifications.
• Degree in any graduate field.

Preferred Skills

• Experience with AWS EMR.

Key Skills

Education

Any Graduate