Description

Build and maintain large-scale data pipelines within the Hadoop ecosystem.

Responsibilities

• Design and develop data pipelines using Hadoop, Spark, and Python or Scala.
• Ingest data from message queues, file shares, REST APIs, and relational databases.
• Optimize Spark, MapReduce, and SQL jobs for performance.
• Transform data from JSON, CSV, and XML formats into usable structures.
• Deploy and manage applications using Jenkins and handle Jar management.

Required Skills

• 5+ years of experience in big data development.
• Extensive experience with the Hadoop ecosystem, including Spark, Sqoop, Kafka, and MapReduce.
• Proficiency in Python or Scala for data processing and scripting.
• Strong skills in Spark SQL and Spark Structured Streaming for real-time processing.
• Proficiency in SQL using MySQL, Hive, or Impala.
• Hands-on experience with NoSQL databases such as HBase or Solr.
• Experience ingesting data from various sources and handling multiple file formats.
• Experience working in a Linux environment.
• Proficiency with version control systems such as Git, SVN, or Bitbucket.

Preferred Skills

• Familiarity with data visualization tools like Elasticsearch and Kibana.

Key Skills

Education

Any Graduate