Description

You will design and build end-to-end batch and near-real-time data pipelines within a distributed systems environment.

This role is on-site.

Responsibilities

• Build data ingestion pipelines from message queues, file shares, REST APIs, and relational databases.
• Design and implement data processing workflows using Spark Structured Streaming and MapReduce.
• Perform performance tuning for Spark, MapReduce, and SQL jobs to optimize data processing.
• Manage source code and version control using Git, SVN, or Bitbucket.
• Collaborate within an Agile development process throughout the full software development life cycle.

Required Skills

• Extensive experience with Hadoop, Spark, and SparkSQL.
• Proficiency in Python, Scala, PHP, or Shell Scripting.
• Strong SQL skills including MySQL, Hive, Impala, or SparkSQL.
• Hands-on experience with Kafka, Sqoop, and MapReduce.
• Experience with NoSQL databases such as HBase, Solr, or Elasticsearch.
• Practical knowledge of data formats including JSON, CSV, and XML.
• Proficiency with Linux operating systems and Jenkins for CI/CD and Jar management.
• Experience working with Cloudera, Hortonworks, CDP, or HDP environments.

Preferred Skills

• Experience with distributed systems and large-scale data ingestion.

Key Skills

Education

Any graduate