Description

You will build and manage large-scale data processing systems and pipelines.

Responsibilities

• Develop data processing applications using Spark Core and Hive.
• Design and maintain workflows using Airflow.
• Manage data storage and processing within Hadoop architecture, including HDFS and MapReduce.
• Implement and maintain CI/CD pipelines for data workloads.
• Integrate caching and queuing stacks like Redis and Kafka into data architectures.

Required Skills

• 4+ years of professional experience in data engineering.
• Proficiency in Python or Java programming.
• Deep expertise in Spark Core, Hive, and Airflow.
• Strong knowledge of Hadoop architecture, MapReduce, and HDFS.
• Experience with UNIX shell scripting.
• Hands-on experience with Redis and Kafka.
• Experience defining and using CI/CD pipelines.

Preferred Skills

• Bachelor's degree in a relevant field.

Key Skills

Education

Bachelor's degree