Description

You will build and deploy large-scale big data pipelines within an AWS environment.

Responsibilities

• Develop data processing logic using Python, Scala, Spark SQL, and DataFrames.
• Build and maintain pipelines using Databricks and Data Lake architectures.
• Optimize, troubleshoot, and debug Spark processes to ensure performance.
• Perform SQL development and implement tuning techniques specifically for Redshift.
• Work within an Agile Scrum framework to deliver data solutions.

Required Skills

• 5+ years of professional software engineering experience.
• Hands-on experience with AWS Databricks.
• Proficiency in Spark, PySpark, and Scala.
• Strong development skills in Python and SQL.
• Experience with AWS services including S3, Redshift, EC2, and Lambda.
• Expertise in SQL optimization and tuning techniques.
• Experience developing in notebooks such as Jupyter, Databricks, or Zeppelin.
• Ability to develop and deploy big data pipelines.
• Any Graduate degree.

Key Skills

Education

Any Graduate