Description

Build and maintain scalable data architectures and processing pipelines within an on-premises environment.

You will own the implementation of data pipelines using Hadoop, Impala, Hive, Spark, and NiFi. You will develop performant Spark jobs in Java and compute tiers operating on these technologies. You will review code and mentor junior engineers to improve the quality of batch jobs. You will evaluate new technologies to enhance system performance, scalability, and reliability.

Responsibilities

• Implement scalable data architectures using Hadoop, Impala, Hive, Spark, and NiFi.
• Develop performant Spark jobs using Java.
• Build data pipelines and compute tiers operating on Hadoop, Impala, and Spark.
• Review code and mentor junior engineers to improve the quality of Hadoop and Spark batch jobs.
• Evaluate new technologies to improve system performance, scalability, and reliability.

Required Skills

• 6+ years of experience in data engineering or big data development.
• Strong proficiency with Hadoop and Spark.
• Experience with Impala and Hive.
• Hands-on experience with NiFi.
• Proficiency in Java for developing Spark jobs.
• Experience building and managing on-premises data architectures.
• Background in designing data pipelines and compute tiers.

Preferred Skills

• Ability to conduct critical code reviews for batch processing workflows.

Key Skills

Education

Any graduate