Description

Lead the design and implementation of large-scale data processing systems within the Hadoop ecosystem. You will own the end-to-end development of batch and real-time streaming pipelines.

Responsibilities

• Architect data processing workflows using Spark for batch and streaming workloads.
• Manage data orchestration and scheduling using Airflow and Oozie.
• Optimize data storage and retrieval using Hive, Impala, Ozone, and Iceberg.
• Deploy and manage data solutions within GCP cloud environments.

Required Skills

• 6+ years of experience in the Hadoop Big Data ecosystem.
• Proven hands-on experience in a technical Lead role.
• Proficiency in Spark using Python or Scala for batch and streaming.
• Hands-on experience with Apache Kafka.
• Experience developing data pipelines in GCP.
• Strong expertise in SQL.
• Proficiency in Python and Shell scripting.
• Experience with Hive, Impala, Ozone, and Iceberg.
• Experience with Airflow and Oozie schedulers.

Preferred Skills

• Deep understanding of Hadoop fundamental concepts.
• Experience with Cloudera ecosystem and CI/CD workflows.

Key Skills

Education

Any Graduate