Description

You will design, build, and optimize large-scale data processing pipelines using modern big data technologies and AI tools.

This role is hybrid.

Responsibilities

• Design and maintain scalable data ingestion, transformation, and analysis pipelines using Hadoop, Spark, and SQL.
• Optimize Spark jobs for performance, handling data skew, resource limitations, and petabyte-scale volumes.
• Implement automated unit, integration, and end-to-end testing frameworks to ensure data quality and pipeline reliability.
• Collaborate with data scientists and cross-functional teams to translate business requirements into technical solutions.
• Leverage AI development tools (GitHub Copilot, ChatGPT) to enhance development workflows and code quality.

Required Skills

• 10+ years of experience in distributed systems and big data engineering.
• Strong proficiency in Python, Scala, or Java for object-oriented development.
• Deep expertise in Apache Spark internals, tuning, and architecture.
• Hands-on experience with Hadoop, Hive, and Trino.
• Proficiency in AWS services (S3, EMR, Glue, Lambda, Athena).
• Advanced SQL skills including window functions and complex joins.
• Experience with AI coding assistants and prompt engineering.
• Bachelor's degree in Computer Science or related field.

Preferred Skills

• Master's degree and past experience in the Financial Services industry.
• Experience leading teams through AI adoption and workflow transformation.

Key Skills

Education

Any Graduate