Description

You design, build, and optimize scalable ETL pipelines and real-time data processing systems for high-volume distributed environments.

This role is on-site.

Responsibilities

• Build and optimize batch and real-time ETL pipelines using PySpark and Apache Spark.
• Develop streaming frameworks with Kafka for low-latency data ingestion and analytics.
• Ensure data quality, governance, and performance optimization across distributed databases.
• Collaborate with cross-functional teams to deliver data-driven insights.
• Improve system scalability, reliability, and cost efficiency.

Required Skills

• 5+ years of experience as a Data Engineer.
• Expertise in PySpark, Apache Spark (Core, SQL, Structured Streaming).
• Strong experience with Kafka (Confluent/Apache) for real-time data ingestion.
• Hands-on experience with Informatica or similar ETL tools.
• Experience with Cloudera Hadoop ecosystem and distributed databases (SingleStore preferred).
• Strong programming skills in Python, Scala, or SQL.
• Familiarity with workflow orchestration tools like Airflow or Oozie.
• Experience handling large-scale data processing (~40TB data, ~5TB daily ingestion).

Preferred Skills

• Knowledge of Medallion Architecture (Bronze, Silver, Gold layers).
• Experience in BFSI or Telecom domains.

Key Skills

Education

Any Graduate