Description

You will build and maintain distributed data processing pipelines using Spark and AWS services.

Responsibilities

• Build Spark Streaming processes and manage Hadoop clusters and their associated services.
• Design, install, configure, and support Hadoop environments.
• Develop High-Level Design (HLD) and Low-Level Design (LLD) mapping specifications.
• Integrate with internal Archival Service Platforms for data purging and lifecycle management.
• Collaborate with data engineering teams to improve integration pipelines and adapt to business needs.

Required Skills

• 5+ years of experience in data engineering or related roles.
• Proficiency in Spark and Spark Streaming.
• Hands-on experience with Hadoop cluster management.
• Experience with Kafka and messaging systems.
• Knowledge of NoSQL databases.
• Experience using AWS services including S3, Athena, and Glue.
• Strong understanding of distributed computing principles.
• Ability to work with internal SDKs and tools.
• Experience performing analysis on large-scale data stores.

Preferred Skills

• Understanding of cloud technology and data lifecycle management.

Key Skills

Education

Any Graduate