Description

Design, develop, and maintain large-scale Spark applications and streaming data pipelines.

This role is on-site.

Responsibilities

• Build and operate streaming-heavy data pipelines using Kafka and Spark Structured Streaming.
• Implement stateful streaming patterns including windowing, watermarking, and checkpointing.
• Develop ingestion and routing flows using Apache NiFi and Kafka-based patterns.
• Optimize Spark jobs through partitioning, memory tuning, shuffle optimization, and efficient data formats.
• Ensure data quality, consistency, and auditability through validation and metadata capture.

Required Skills

• 5+ years of hands-on experience with Apache Spark in production environments.
• Advanced proficiency in Scala and PySpark.
• Strong experience with Kafka-based streaming architectures.
• Solid understanding of distributed systems and data processing at scale.
• Experience building batch and real-time pipelines.
• Proficiency in SQL, Linux, Git, and CI/CD.

Preferred Skills

• Experience integrating Spark with distributed object storage systems like Apache Ozone and Ceph.
• Knowledge of ETL/ELT pipeline design with low latency and fault tolerance.

Key Skills

Education

Bachelor's degree