Description

You will design and implement big data engineering projects within the Hadoop ecosystem. You own the build and maintenance of distributed, scalable data pipelines for real-time and batch processing.

Responsibilities

• Build distributed data pipelines using Spark Streaming and Hive for real-time ingestion and processing.
• Engineer batch and streaming solutions on Cloudera, MapR, or HDP platforms.
• Implement ETL processes with Apache NiFi to integrate data from multiple sources.
• Manage Hadoop clusters and services including Hive, HBase, MapReduce, and Sqoop.
• Optimize analytics performance using HDFS formats like Parquet and Avro.

Required Skills

• Hands-on experience with the Hadoop ecosystem: Hive, Pig, HBase, Spark.
• Proficiency in Java, Python, or Scala.
• Experience managing Hadoop jobs using a scheduler.
• Ability to work with XML, JSON, ORC, and CSV data formats.
• Knowledge of NoSQL databases: HBase, Cassandra, MongoDB.
• Experience cleaning data via streaming APIs or user-defined functions.
• Ability to assess dataset quality for a Hadoop data lake.
• Bachelor's Degree required.

Preferred Skills

• Fine-tuning Hadoop applications for high performance and troubleshooting runtime issues.

Key Skills

Education

Bachelor's Degree