Description

You will design and build end-to-end near-real-time and batch data pipelines.

This role is on-site.

Responsibilities

• Develop data ingestion pipelines from message queues, file shares, REST APIs, and relational databases.
• Build and maintain streaming solutions using Spark Structured Streaming.
• Tune performance for Spark, MapReduce, and SQL jobs.
• Manage source code, version control, and Jenkins CI/CD processes.
• Resolve complex analytical problems in an Agile environment.

Required Skills

• 5+ years of experience with Hadoop, Big Data, and Distributed Systems.
• Proficiency in Scala, Python, or PHP for programming tasks.
• Strong SQL skills with MySQL, Hive, Impala, or Spark SQL.
• Hands-on experience with Kafka, Sqoop, and NoSQL databases like HBase or Solr.
• Experience with Elastic Search and Kibana for monitoring and logging.
• Deep understanding of Linux operating systems.
• Familiarity with Cloudera or Hortonworks (CDP/HDP) ecosystems.
• Experience with Git, Bitbucket, and Jenkins.

Preferred Skills

• Experience with MapReduce and legacy batch processing.
• Proficiency in Shell Scripting.

Key Skills

Education

Any Gradute