Description

You will build and maintain enterprise-scale data pipelines on AWS, optimizing Spark and SQL workloads for performance and cost.

This role is hybrid.

Responsibilities

• Design and maintain scalable big data pipelines using Spark, Hadoop, and Hive.
• Optimize complex SQL transformations, window functions, and multi-joins for efficiency.
• Implement automated testing, CI/CD pipelines, and data quality validation.
• Deploy and manage AWS data services including EMR, Glue, Athena, and S3.
• Leverage AI-assisted tools to improve development productivity and resolve bottlenecks.

Required Skills

• 5+ years building data solutions with Spark and Hadoop ecosystem.
• Expert-level SQL with complex joins and window functions.
• Strong scripting in Python or Perl.
• Scala programming experience.
• AWS cloud experience (S3, EMR, Glue, Athena).
• Experience with CI/CD, automated testing, and GitHub workflows.
• Bachelor's degree in a technical field.

Preferred Skills

• Experience in financial services or regulated industries.
• Proficiency in Agile delivery methods.

Key Skills

Education

Bachelor's degree