Description

You will design, develop, and maintain large-scale data processing pipelines using distributed systems and cloud platforms. This role owns the end-to-end lifecycle of data solutions, from ingestion and transformation to optimization and testing.

This role is hybrid.

Responsibilities

• Build and maintain scalable data pipelines using Hadoop, Spark, and Python.
• Optimize Spark jobs for performance, handling data skew, petabyte-scale volumes, and resource limitations.
• Design and implement automated testing frameworks for data quality assurance and CI/CD.
• Collaborate with data scientists to translate business requirements into technical data solutions.
• Maintain and troubleshoot production data pipelines to ensure reliability and accuracy.

Required Skills

• 5+ years of experience with Big Data technologies: Hadoop, Spark, Hive, Trino.
• Proficiency in Python, Scala, and SQL (window functions, complex joins, aggregations).
• Hands-on experience with AWS services (S3, EMR, Glue, Lambda, Athena).
• Strong understanding of Spark internals: executors, tasks, stages, DAG, partitioning, caching, and broadcast joins.
• Experience with Agile methodologies (Scrum, Kanban) and iterative development.
• Ability to write and maintain automated unit, integration, and end-to-end tests.
• Strong object-oriented programming skills and database concepts.

Preferred Skills

• Experience with AI development tools (GitHub Copilot, ChatGPT, Claude) and prompt engineering.
• Experience redesigning development workflows to leverage AI capabilities.
• Master's degree or prior experience in the Financial Services industry.

Key Skills

Education

Any Graduate