Description

You will design and maintain scalable data pipelines across on-prem Hadoop ecosystems and AWS cloud platforms.

This role is on-site.

Responsibilities

• Develop and optimize Apache Spark and PySpark jobs for large-scale batch and streaming workloads.
• Build and maintain Java-based applications, utilities, and internal data processing libraries using Gradle.
• Orchestrate workflows and schedule jobs using Apache Airflow DAGs.
• Design and support Kafka-based streaming pipelines for real-time data ingestion.
• Implement data quality checks, monitoring, and alerting using Prometheus and Grafana.

Required Skills

• 10+ years of experience in data engineering or related backend roles.
• Strong proficiency in Java application development and build tools.
• Hands-on experience with AWS services: S3, Glue, EMR, and Athena.
• Deep knowledge of the Hadoop ecosystem, including HDFS and Hive.
• Experience building and managing Apache Spark and PySpark pipelines.
• Proficiency in Apache Kafka producers, consumers, and topic management.
• Strong SQL skills and experience with Python for data processing.
• Familiarity with Git and CI/CD pipelines.
• Experience with observability tools like Prometheus and Grafana.

Preferred Skills

• Experience with Apache Iceberg table formats.
• Exposure to Kubernetes (EKS) and hybrid on-prem/cloud architectures.

Key Skills

Education

Any Graduate