Description

You will design, develop, and maintain large-scale Spark applications and streaming data pipelines.

This role is on-site.

Responsibilities

• Build and operate streaming-heavy data pipelines using Kafka and Spark Structured Streaming.
• Implement stateful streaming patterns, including windowing, watermarking, and late data handling.
• Develop event replay and reprocessing workflows using Kafka offsets and partitions.
• Build ingestion and routing flows using Apache NiFi and Kafka.
• Optimize Spark jobs through partitioning, memory tuning, shuffle optimization, and efficient data formats.

Required Skills

• 5+ years of experience with Big Data technologies.
• Strong proficiency in Scala and Python.
• Deep expertise with Apache Spark (PySpark) and distributed computing.
• Hands-on experience with Apache Kafka for streaming and event processing.
• Experience with Linux environments and Git version control.
• Familiarity with CI/CD pipelines for data engineering workflows.
• Bachelor's degree in Computer Science or related field.

Key Skills

Education

Bachelor's degree