Description

You will design, develop, and maintain large-scale Spark applications and streaming-heavy data pipelines.

This role is on-site.

Responsibilities

• Build and operate real-time data pipelines using Kafka and Spark Structured Streaming.
• Implement stateful streaming patterns including windowing, watermarking, and late data handling.
• Develop event replay and reprocessing workflows using Kafka offsets and partitions.
• Optimize Spark jobs through partitioning, memory tuning, and shuffle optimization.
• Ensure data quality, consistency, and auditability through validation and reconciliation.

Required Skills

• 7+ years of hands-on experience with Apache Spark in production environments.
• Advanced proficiency in Scala and PySpark.
• Strong experience with Kafka-based streaming architectures and event ingestion.
• Hands-on experience with Apache NiFi for data ingestion and flow management.
• Proficiency in SQL, Linux, shell scripting, and Git-based version control.
• Experience building batch and real-time ETL/ELT pipelines.
• Solid understanding of distributed systems and data processing at scale.
• Experience with distributed object storage systems such as Apache Ozone or Ceph.

Preferred Skills

• Bachelor’s degree in Computer Science, Engineering, or equivalent practical experience.
• Experience contributing to reusable frameworks and engineering best practices.

Key Skills

Education

Bachelor's degree