Description

You will build and manage data pipelines and ETL processes within a Hadoop ecosystem.

Responsibilities

• Develop data pipelines to ingest data and implement ETL using NiFi, Kafka, Spark, and Hive.
• Build storage and computation frameworks using AWS services including S3, EC2, Lambda, EMR, and Redshift.
• Create and maintain mapping logic to integrate Hadoop datasets from various source systems.
• Automate data migration processes using Sqoop and Oozie.

Required Skills

• Hands-on experience with Apache NiFi and Kafka.
• Proficiency in Spark and Hive for data processing.
• Experience with AWS services: S3, EC2, Lambda, EMR, and Redshift.
• Ability to use Sqoop and Oozie for data movement and workflow orchestration.
• Strong Shell scripting skills for task automation.
• Experience designing software systems to predict and measure design outcomes.

Preferred Skills

• Master’s degree in Computer Science or Engineering with 1 year of experience, or a Bachelor’s degree with 5 years of progressive experience.

Key Skills

Education

Any graduate