Description

You design, develop, and maintain Hadoop-based data processing solutions for large-scale enterprise environments.

Responsibilities

• Manage and optimize Hadoop clusters, including HDFS, YAARN, and related components.
• Develop and support batch and real-time data pipelines using Hadoop ecosystem tools.
• Implement performance tuning, capacity planning, and resource optimization for Hadoop clusters.
• Collaborate with data architects and application teams to design scalable data architectures.
• Troubleshoot and resolve issues related to data ingestion, processing, and storage.

Required Skills

• 12+ years of experience in data engineering or big data.
• Strong hands-on experience with Hadoop and components like HDFS, MapReduce, Hive, Pig, and HBase.
• Extensive experience with data ingestion tools such as Sqoop, Flume, or Kafka.
• Proficiency in Java, Scala, Python, or SQL.
• Experience building and optimizing large-scale batch and streaming data pipelines.
• Strong understanding of distributed systems, data partitioning, and fault tolerance.
• Experience working in Linux/Unix environments.
• Knowledge of data modeling, ETL processes, and data warehousing concepts.

Preferred Skills

• Experience with Spark (Core, SQL, Streaming).
• Experience with NoSQL databases such as MongoDB.
• Familiarity with workflow orchestration tools like Apache Airflow or Oozie.

Key Skills

Education

Bachelor's degree