Description

You will engineer and manage data pipelines on Google Cloud Platform.

This role is remote.

Responsibilities

• Design, implement, and manage data pipelines using Apache NiFi, Dataproc, BigQuery, Dataflow, and other GCP services.
• Develop Spark and Hadoop workflows within Dataproc to process large datasets, ensuring reliability and efficiency.
• Integrate Dataproc workflows with GCP services including Cloud Storage, BigQuery, and Pub/Sub.
• Troubleshoot data quality issues and optimize pipeline performance within Dataproc clusters.
• Document data pipelines, ETL processes, and overall data architecture.

Required Skills

• 5+ years of professional experience in data engineering.
• Expert proficiency in SQL.
• Hands-on experience with Python, Spark, and Scala for data processing.
• Proven experience building data workflows using Apache NiFi.
• Strong familiarity with GCP services, specifically Dataproc, BigQuery, and Dataflow.
• Experience working with Spark and Hadoop environments.
• Ability to integrate data services like Cloud Storage and Pub/Sub.

Preferred Skills

• Experience optimizing large-scale data architectures.
• Background in ETL process documentation and data quality assurance.

Key Skills

Education

Any Gradute