Description

Design, develop, and maintain large-scale data processing pipelines using distributed systems and cloud platforms.

This role is hybrid.

Responsibilities

• Build scalable data ingestion, storage, and transformation solutions using Hadoop, Spark, and SQL.
• Optimize pipeline performance, scalability, and reliability for petabyte-scale datasets.
• Develop automated testing frameworks for data quality assurance and pipeline robustness.
• Troubleshoot production issues including data skew, resource limitations, and job failures.
• Collaborate with data scientists to translate business requirements into technical data solutions.

Required Skills

• 5+ years of experience in distributed systems and big data technologies.
• Proficiency in Python, Scala, or Java for object-oriented development.
• Deep understanding of Apache Spark internals, DAGs, executors, and performance tuning (partitioning, caching, broadcast joins).
• Strong SQL skills including window functions, complex joins, and aggregation.
• Hands-on experience with AWS services (S3, EMR, Glue, Lambda, Athena).
• Experience with Test Automation, Build Automation, and Configuration Management frameworks.
• Proficiency in AI development tools (GitHub Copilot, ChatGPT, Claude) and prompt engineering for coding assistance.
• Experience working in Agile/Scrum environments with iterative delivery.

Preferred Skills

• Master’s degree in Computer Science or related discipline.
• Prior experience in the Financial Services industry.

Key Skills

Education

Any Graduate