Description

Design and maintain large-scale data processing pipelines and AI-integrated workflows for petabyte-scale datasets.

This role is hybrid.

Responsibilities

• Build and optimize scalable data ingestion, transformation, and storage pipelines using Hadoop, Spark, and SQL.
• Develop automated testing frameworks for data quality, including unit, integration, and end-to-end tests.
• Troubleshoot production issues related to data skew, resource limitations, and job failures.
• Collaborate with data scientists to translate business requirements into technical solutions and AI-driven insights.
• Optimize Spark jobs using partitioning, caching, and broadcast joins for performance.

Required Skills

• 5+ years of experience with Big Data technologies: Hadoop, Spark, Hive, Trino.
• Proficiency in object-oriented languages: Java, Scala, or Python.
• Strong SQL skills: window functions, complex joins, and aggregation.
• Hands-on experience with AI development tools (GitHub Copilot, ChatGPT, Claude) for prompt engineering and workflow design.
• Cloud exposure to AWS services: S3, EMR, Glue, Lambda, Athena.
• Experience with CI/CD, test automation, and Agile/Scrum methodologies.
• Ability to handle petabyte-scale data volumes and mitigate data skew.

Preferred Skills

• Master's degree in Computer Science or related discipline.
• Past experience in the Financial Services industry.

Key Skills

Education

Any Graduate