Description

You will design and implement data processing frameworks and pipelines within the Google Cloud Platform ecosystem.

This role is remote.

Responsibilities

• Build high-performing data processing frameworks leveraging GCP services.
• Develop batch and real-time streaming data pipelines for collection, storage, and transformation.
• Perform ETL and data engineering tasks using Dataflow, Dataproc, and BigQuery.
• Schedule and orchestrate complex data workflows using Airflow and Cloud Composer.

Required Skills

• 8+ years of experience in data engineering.
• Hands-on experience with GCP services including BigQuery, Dataflow, Cloud Storage, Dataproc, Pub/Sub, and Cloud Composer.
• Proficiency in Python and Shell scripting.
• Experience with Spark (Batch & Streaming) using Python or Scala.
• Practical knowledge of Apache Kafka.
• Programming experience in Java, Python, or Scala.
• Experience with Airflow or Cloud Composer for orchestration.
• Working knowledge of BigQuery, BigTable, or Teradata.
• Familiarity with the Hadoop Big Data ecosystem.

Preferred Skills

• Experience with Kubernetes, Docker containers, and deploying GCP services.
• Knowledge of Scrum/Agile methodologies.

Key Skills

Education

Not specified