Description

Design and build end-to-end near-real-time and batch data pipelines within a distributed systems environment.

Responsibilities

• Build and maintain data pipelines using Spark Structured Streaming and MapReduce.
• Ingest data from message queues, file shares, REST APIs, and relational databases.
• Tune the performance of Spark, MapReduce, and SQL jobs.
• Manage code through version control systems and handle Jenkins and Jar management.
• Analyze customer requests to design and implement technical solutions.

Required Skills

• 5+ years of experience in Big Data and distributed systems.
• Proficiency in SQL using MySQL, Hive, Impala, or Spark SQL.
• Hands-on experience with Spark, Sqoop, Kafka, and MapReduce.
• Experience with NoSQL databases such as HBase, Solr, or Elasticsearch.
• Strong programming skills in Scala, Python, PHP, or Shell scripting.
• Experience handling data formats including JSON, CSV, and XML.
• Deep understanding of the SDLC and Agile development processes.
• Proficiency with Linux operating systems.
• Experience with version control tools like Git, SVN, or Bitbucket.

Key Skills

Education

Any Graduate