Description

You will design, build, and maintain large-scale data processing pipelines using distributed systems and cloud platforms.

This role is hybrid.

Responsibilities

• Design and develop scalable data ingestion, transformation, and analysis solutions using Hadoop, Spark, and cloud services.
• Optimize existing pipelines for performance, scalability, and reliability, addressing issues like data skew and resource limitations.
• Develop and maintain automated testing frameworks for data quality assurance, including unit, integration, and end-to-end tests.
• Monitor and troubleshoot production data pipelines to resolve job failures and ensure system stability.
• Collaborate with data scientists and analysts to translate business requirements into technical solutions and support data-driven decisions.

Required Skills

• 5+ years of experience in software engineering with a focus on Big Data technologies.
• Strong proficiency in Python, Scala, or Java for object-oriented development.
• Deep expertise in Apache Spark, including core architecture (executors, tasks, DAGs) and performance tuning (partitioning, caching, broadcast joins).
• Hands-on experience with Hadoop ecosystem components such as Hive and Trino.
• Proficiency in SQL, including complex queries, window functions, and multi-table joins.
• Experience with cloud platforms, specifically AWS services like S3, EMR, Glue, Lambda, and Athena.
• Strong background in CI/CD, build automation, and configuration management frameworks.

Preferred Skills

• Hands-on experience with AI development tools (GitHub Copilot, ChatGPT, Claude) and prompt engineering for coding assistance.
• Experience redesigning development workflows to leverage AI capabilities for efficiency.
• Past experience in the Financial Services industry.

Key Skills

Education

Any Graduate