Description

You will architect and maintain large-scale data pipelines, ensuring scalability and performance for big data platforms.

This role is hybrid.

Responsibilities

• Design and develop distributed data processing pipelines using Hadoop, Spark, and Python/Scala.
• Implement data ingestion, storage, and transformation solutions that handle petabyte-scale volumes.
• Optimize Spark jobs by tuning executors, stages, and DAGs to resolve resource limitations and data skew.
• Develop and maintain automated testing frameworks for data quality, unit, and integration testing.
• Collaborate with cross-functional teams to translate business requirements into technical data solutions.

Required Skills

• 5+ years of experience with Big Data technologies: Hadoop, Spark, Hive, and Trino.
• Strong proficiency in object-oriented languages: Java, Scala, or Python.
• Advanced SQL skills including window functions, complex joins, and query optimization.
• Deep understanding of Spark internals, architecture, and performance tuning techniques.
• Experience with Agile methodologies: Scrum, Kanban, and continuous improvement.
• Proficiency with AI development tools (GitHub Copilot, ChatGPT, Claude) and prompt engineering.
• Ability to troubleshoot production issues related to data skew, bad data, and scalability.

Preferred Skills

• Master's degree and experience in the Financial Services industry.
• Experience leading teams through AI adoption and workflow transformation.

Referral Bonus: ₹50,000

Key Skills

Education

Any Graduate