Description

Design, develop, and maintain large-scale Spark applications and streaming data pipelines for production systems.

This role is on-site.

Responsibilities

• Build and operate streaming-heavy data pipelines using Kafka and Spark Structured Streaming, implementing stateful patterns like windowing and watermarking.
• Develop ingestion and routing flows using Apache NiFi, including Kafka-based ingestion patterns and event replay workflows.
• Optimize Spark jobs through partitioning strategies, memory tuning, shuffle optimization, and efficient data formats.
• Integrate Spark workloads with distributed object storage systems such as Apache Ozone and Ceph.
• Ensure data quality and consistency through validation, reconciliation, and metadata capture while supporting production monitoring and incident resolution.

Required Skills

• 5+ years of hands-on experience with Apache Spark in production environments.
• Advanced proficiency in Scala and PySpark for building batch and real-time pipelines.
• Strong experience with Kafka-based streaming architectures and Spark Structured Streaming.
• Proficiency with Linux, shell scripting, and Git-based version control.
• Strong SQL skills and experience working with structured and semi-structured data.
• Experience building robust event replay and reprocessing workflows using Kafka offsets and partitions.
• Bachelor’s degree in Computer Science, Engineering, or equivalent practical experience.

Preferred Skills

• Experience with Apache Ozone and/or Ceph as storage backends for analytics workloads.
• Strong background in Spark performance tuning (CPU, memory, I/O, shuffle).
• Familiarity with CI/CD pipelines and automated testing for data applications.

Key Skills

Education

Bachelor's degree