Description

Job Responsibilities include:

Design, develop, and maintain large-scale batch and real-time data pipelines using Apache Spark, Scala, and PySpark.

Build and manage streaming data architectures using Kafka and Spark Structured Streaming.

Develop data ingestion and orchestration workflows using Apache NiFi for efficient data flow management.

Optimize Spark jobs for performance, scalability, and reliability through tuning and efficient data handling.

Ensure data quality, validation, and consistency across pipelines and storage systems.

Collaborate with cross-functional teams and support production systems through monitoring, troubleshooting, and continuous improvement.

Required Qualifications:

Bachelor’s degree in computer science, Engineering, or a related field (or equivalent practical experience).

8–10+ years of experience in data engineering with strong focus on big data technologies.

Strong hands-on expertise in Apache Spark (Scala/PySpark), Kafka, and real-time streaming architectures.

Proven experience in building scalable data pipelines using Apache NiFi, SQL, and distributed storage systems like Ozone or Ceph

Key Skills

Education

Any Graduate