Description

You will manage and optimize data processing pipelines within a GCP environment.

Responsibilities

• Engineer data pipelines using Scala and Spark for both streaming and batch processing.
• Build and maintain CI/CD workflows using GitLab pipelines.
• Troubleshoot Spark jobs and resolve performance issues using the Spark UI.
• Deploy and manage data infrastructure on GCP.

Required Skills

• 5+ years of professional experience in data engineering.
• Strong proficiency in Scala and Spark (batch and streaming).
• Experience with PySpark.
• Hands-on experience with GitLab pipelines and CI/CD concepts.
• Practical knowledge of GCP.
• Ability to troubleshoot complex Spark jobs via Spark UI.
• Immediate ability to contribute to production environments without training.
• Any graduate degree.

Key Skills

Education

Any Graduate