Description

Design and build near-real-time and batch data pipelines for distributed systems.

This role is on-site.

Responsibilities

• Develop data ingestion pipelines from message queues, file shares, REST APIs, and relational databases.
• Implement streaming solutions using Spark Structured Streaming.
• Optimize performance for Spark, MapReduce, and SQL jobs.
• Manage end-to-end design and build processes for data solutions.
• Utilize version control systems like Git and Jenkins for Jar management.

Required Skills

• 5+ years of experience with Hadoop, Big Data, and Distributed Systems.
• Strong SQL proficiency in MySQL, Hive, Impala, or Spark SQL.
• Hands-on programming in Scala or Python.
• Experience with Kafka, Sqoop, and NoSQL databases like HBase or Solr.
• Proficiency with Linux operating systems.
• Knowledge of data formats including JSON, CSV, and XML.
• Familiarity with Agile development and SDLC phases.

Preferred Skills

• Experience with Cloudera (CDP) or Hortonworks (HDP) distributions.
• Proficiency with Elastic Search and Kibana.

Key Skills

Education

Any Gradute