Description

Develop and optimize data processing applications within a distributed big data environment.

You will own the release pipeline.

Responsibilities

• Build data ingestion pipelines from message queues, file shares, REST APIs, and relational databases.
• Implement Spark Structured Streaming for real-time data processing.
• Perform performance tuning for Spark, MapReduce, and SQL jobs.
• Manage data formats including JSON, CSV, and XML.
• Operate within Linux-based distributed systems and Hadoop ecosystems.

Required Skills

• 5+ years of professional programming experience.
• Strong SQL skills with MySQL, Hive, Impala, or Spark SQL.
• Hands-on programming in Scala, Python, PHP, or Shell Scripting.
• Experience with Kafka, Sqoop, and MapReduce.
• Proficiency with NoSQL databases such as HBase, Solr, or Elasticsearch.
• Working knowledge of Hadoop/Big Data ecosystems (Cloudera, Hortonworks, CDP, or HDP).
• Deep proficiency with the Linux operating system.
• Experience handling data ingestion via REST APIs and message queues.

Preferred Skills

• Experience with Spark Structured Streaming.

Key Skills

Education

ANY GRADUATE