Description

You will design and maintain end-to-end big data pipelines using Hadoop and Spark, handling both near-real-time and batch processing.

This role is hybrid.

Responsibilities

• Build and maintain data pipelines ingesting from message queues, file shares, REST APIs, and relational databases.
• Tune performance for Spark, MapReduce, and SQL jobs to ensure efficiency and reliability.
• Resolve technical issues and develop solutions for customer requests.
• Manage data ingestion across formats including JSON, CSV, and XML.

Required Skills

• 7+ years of professional experience in data engineering.
• Expertise in Hadoop and distributed systems.
• Proficiency in Apache Spark, including Spark Structured Streaming.
• Strong SQL skills using Hive, Impala, or MySQL.
• Hands-on programming experience in Scala or Python.
• Experience with version control systems like Git or SVN.
• Working knowledge of the Linux operating system.

Preferred Skills

• Familiarity with Agile development and SDLC phases.

Key Skills

Education

Any Graduate