Description

You will manage and enhance data pipelines and large-scale datasets within a Hadoop environment.

Responsibilities

• Support existing data ingestion jobs and maintain pipeline stability.
• Create, support, and schedule new data ingestions from diverse sources.
• Perform ongoing data engineering and enhancements to datasets.
• Build and maintain real-time streaming datasets.
• Analyze large datasets ranging from 10 to 1000 TB.

Required Skills

• 5+ years of experience in big data engineering.
• Proficiency with Hive, Pig, HBase, and MapReduce.
• Hands-on experience with Sqoop, Flume, Spark Streaming, Kafka, and NiFi.
• Experience with traditional databases such as Netezza, MySQL, Teradata, or Oracle.
• Familiarity with data science toolkits including R, Jupyter, and Python.
• Bachelor's degree in Computer Science.

Preferred Skills

• Experience with Azure products including Azure Data Lake Store, Azure HD Insights, Cosmos DB, and PowerBI.

Key Skills

Education

Bachelor's degree in Computer Science