Description

You will build and maintain big data pipelines within an AWS environment.

Responsibilities

• Develop and deploy big data pipelines using PySpark and Spark SQL.
• Build data processing logic using Python, Scala, and Spark DataFrames.
• Optimize and tune SQL queries specifically for Redshift.
• Troubleshoot, debug, and perform performance tuning on Spark processes.
• Work within an Agile Scrum framework to deliver data solutions.

Required Skills

• 5+ years of professional software engineering experience.
• Hands-on experience with Databricks and PySpark.
• Proficiency in Python and SQL development.
• Experience with AWS services including S3, Redshift, EC2, and Lambda.
• Strong understanding of Data Lake architectures.
• Experience developing in Notebook environments like Jupyter, Databricks, or Zeppelin.
• Practical knowledge of Spark optimization and tuning techniques.
• Bachelor's degree or equivalent graduate qualification.

Key Skills

Education

Any Graduate