Description

You will design, develop, and maintain large-scale batch and real-time data pipelines using Apache Spark, Scala, and PySpark. This role is on-site.

Responsibilities

• Build and manage streaming data architectures using Kafka and Spark Structured Streaming.
• Develop data ingestion and orchestration workflows using Apache NiFi for efficient data flow management.
• Optimize Spark jobs for performance, scalability, and reliability through tuning and efficient data handling.
• Ensure data quality, validation, and consistency across pipelines and storage systems.
• Collaborate with cross-functional teams and support production systems through monitoring, troubleshooting, and continuous improvement.

Required Skills

• 8+ years of experience in data engineering with a strong focus on big data technologies.
• Strong hands-on expertise in Apache Spark (Scala/PySpark).
• Proven experience with Kafka and real-time streaming architectures.
• Experience building scalable data pipelines using Apache NiFi.
• Proficiency in SQL and distributed storage systems like Ozone or Ceph.
• Bachelor’s degree in computer science, Engineering, or a related field (or equivalent practical experience).

Preferred Skills

• Experience with data quality validation tools.
• Familiarity with production monitoring and troubleshooting in distributed environments.

Key Skills

Education

Any Graduate