Description

Build and maintain scalable data processing pipelines on AWS using Python, Spark, and Hadoop/EMR.

This role is on-site.

Responsibilities

• Develop efficient data processing pipelines using Python, Spark, Hive, and Hadoop/EMR.
• Implement and manage AWS services including Lambda, EventBridge, Step Functions, Redshift, S3, and Glue.
• Execute SQL queries, data manipulation, and analysis to extract meaningful insights from large datasets.
• Build and maintain CI/CD pipelines using Gitlab and Terraform.
• Collaborate with team members to solve complex data problems and drive solutions.

Required Skills

• 10+ years of hands-on Python development experience for big data applications.
• 5+ years of development experience with AWS platforms (EMR, Lambda, EventBridge, Step Functions, Redshift, S3, Glue).
• 5+ years of experience making API calls, writing SQL queries, and using Spark for data transformation.
• Hands-on experience with CI/CD pipeline tools such as Gitlab and Terraform.
• Strong analytical and problem-solving skills.
• Bachelor's degree in a relevant field.

Preferred Skills

• Experience with data visualization and mining.
• Ability to drive alternative plans and conclusions.

Key Skills

Education

Bachelor's degree