Description

You will design and develop large-scale, fault-tolerant data pipelines using Hadoop and related technologies.

Responsibilities

• Optimize data workflows for performance and scalability.
• Collaborate with data scientists, analysts, and stakeholders to understand data needs.
• Maintain data quality, security, and governance standards.
• Analyze existing data architecture and recommend improvements.
• Automate data extraction, transformation, and loading processes.

Required Skills

• 5+ years of professional experience in data engineering within Hadoop ecosystems.
• Proficiency in Hadoop ecosystem components: HDFS, MapReduce.
• Experience with Apache Spark, Hive, and Pig.
• Strong skills in programming languages: Python, Scala, Java.
• Familiarity with ETL tools and pipelines.
• Knowledge of SQL and NoSQL databases (HBase, Cassandra).
• Experience with Kafka.
• Proficiency with Git for version control.

Key Skills

Education

Bachelor's or Master's degrees