Description

You will engineer and manage data pipelines on Google Cloud Platform.

Responsibilities

• Design, implement, and manage data pipelines using Apache NiFi, Dataproc, BigQuery, Dataflow, and other GCP services.
• Develop Spark and Hadoop workflows within Dataproc to process large datasets, ensuring reliability and efficiency.
• Integrate Dataproc workflows with GCP services including Cloud Storage, BigQuery, and Pub/Sub.
• Troubleshoot data quality issues and optimize pipeline performance within Dataproc clusters.
• Document data pipelines, ETL processes, and overall data architecture.

Required Skills

• 5+ years of professional experience.
• Expert proficiency in SQL and Python/Spark/Scala for data processing.
• Hands-on experience with GCP services (Dataproc, BigQuery, Dataflow).
• Experience building data workflows using Apache NiFi.
• Familiarity with Spark and Hadoop environments.
• Ability to integrate data services like Cloud Storage and Pub/Sub.

Key Skills

Education

Any Gradute