Description

You will build and maintain scalable data pipelines and processing systems within the Hadoop ecosystem.

Responsibilities

• Design and develop scalable data pipelines for large-scale data processing.
• Execute data ingestion using Kafka, Spark, and Sqoop across various file formats.
• Process data into Hive using Beeline or Spark and orchestrate workflows using Apache Airflow.
• Automate data processing tasks through shell scripting and implement full and incremental data loading strategies.
• Manage version control using Git.

Required Skills

• 5+ years of experience in Data Engineering, ETL, and data warehousing.
• Strong knowledge of the Hadoop ecosystem (HDFS, YARN, MapReduce, Tez, Spark).
• Proficiency with Kafka, Spark, Sqoop, and Hive.
• Expertise in full and incremental data loading techniques to ensure consistency.
• Experience with shell scripting for automation.
• Any Graduate degree.

Preferred Skills

• Understanding of PySpark for real-time data processing.
• Hands-on experience with Apache Airflow.
• Experience with Postgres SQL, SQL Server, or MSBI.

Key Skills

Education

Any Graduate