Description

You will build and manage large-scale big data pipelines within an AWS environment.

This role is on-site.

Responsibilities

• Develop and deploy big data pipelines using PySpark and Spark SQL.
• Build and maintain data processes using Databricks, Data Lake, and Python.
• Optimize and tune SQL queries specifically for Redshift performance.
• Troubleshoot, debug, and perform performance tuning on Spark processes.
• Work within an Agile Scrum framework to deliver technical requirements.

Required Skills

• 5+ years of experience in software engineering or data engineering.
• Hands-on expertise with AWS, Databricks, and PySpark.
• Proficiency in Python and Spark development.
• Strong SQL development skills including optimization and tuning techniques.
• Experience with AWS services including S3, Redshift, EC2, and Lambda.
• Practical experience working in Notebook environments such as Jupyter, Databricks, or Zeppelin.
• Solid understanding of Spark DataFrames and Spark SQL.

Preferred Skills

• Bachelor's degree or equivalent graduate qualification.

Key Skills

Education

Any Graduate