Description

Own the development and maintenance of large-scale data pipelines using Hadoop ecosystem tools.

Responsibilities

• Design and implement data processing workflows using Spark Structured Streaming and Kafka.
• Write and optimize complex queries in SQL and Python for data extraction and transformation.
• Manage data storage and retrieval using Hive and HBase.
• Parse and handle structured data formats including JSON, CSV, and XML.
• Deploy and manage code via Git and Jenkins within an Agile environment on Linux.

Required Skills

• 6+ years of hands-on experience with Hadoop and Big Data technologies.
• Strong proficiency in Python programming.
• Advanced SQL skills.
• Experience with Spark Structured Streaming.
• Working knowledge of Kafka for stream processing.
• Familiarity with Hive and HBase.
• Experience with Linux command line tools.
• Proficiency with Git for version control.
• Experience with Jenkins for CI/CD pipelines.

Preferred Skills

• Experience working with Agile methodologies.
• Graduate degree in any discipline.

Key Skills

Education

Any Gradute