Description

You will build and maintain scalable data pipelines and processing systems within the Hadoop ecosystem.

Responsibilities

• Design and develop scalable data pipelines for large-scale data processing.
• Execute data ingestion using Kafka, Spark, and Sqoop across various file formats.
• Process data into Hive using Beeline or Spark.
• Orchestrate and monitor workflows using Apache Airflow.
• Automate data processing tasks through shell scripting and implement full and incremental data loading strategies.

Required Skills

• 5+ years of experience in Data Engineering, ETL, and data warehousing.
• Strong knowledge of the Hadoop ecosystem, including HDFS, YARN, MapReduce, Tez, and Spark.
• Proficiency with Kafka, Spark, Sqoop, and Hive.
• Expertise in full and incremental data loading techniques to ensure consistency.
• Experience with shell scripting for automation.
• Proficiency with Git for version control.
• Any Graduate degree.

Preferred Skills

• Understanding of PySpark for real-time data processing.
• Hands-on experience with Apache Airflow.
• Experience with Postgres SQL, SQL Server, or MSBI.

Key Skills

Education

Any Graduate