Description

You will design and build end-to-end near-real-time and batch data pipelines within a distributed systems environment.

Responsibilities

• Design and build end-to-end near-real-time and batch data pipelines.
• Ingest data from message queues, file shares, REST APIs, and relational databases using formats like JSON, CSV, and XML.
• Perform tuning on Spark, MapReduce, or SQL jobs to optimize performance.
• Manage source code and version control using SVN, Git, or Bitbucket.
• Collaborate within an Agile development process across all SDLC phases.

Required Skills

• 5+ years of experience with Hadoop, Big Data, and distributed systems.
• Strong SQL skills including MySQL, Hive, Impala, or Spark SQL.
• Hands-on experience with Spark, Sqoop, Kafka, MapReduce, and Spark Structured Streaming.
• Proficiency in at least one programming language: Scala, Python, PHP, or Shell Scripting.
• Experience with NoSQL databases such as HBase, Solr, Elasticsearch, or Kibana.
• Working knowledge of CDP, HDP, Cloudera, or Hortonworks.
• Proficiency with Linux operating systems.
• Experience with Jenkins and Jar management.
• Degree in any graduate field.

Preferred Skills

• Experience with Elastic Search and Kibana.

Key Skills

Education

Any Graduate