Description

Design, build, and manage data processing systems and pipelines within the Google Cloud Platform environment.

Responsibilities

• Design and manage data processing workflows on GCP using Dataflow, Dataproc, and Cloud Composer.
• Build and maintain automated pipelines to collect, process, and distribute data from traditional and cloud sources.
• Implement CI/CD pipelines for data applications to automate build, package, and deployment processes.
• Manage source code repositories and enforce robust version control practices.
• Monitor and optimize the performance and cost of data infrastructure and Dataflow jobs.

Required Skills

• 3 to 7 years of industry experience in data engineering or cloud environments.
• Hands-on experience with Google Cloud Platform (GCP), including BigQuery, Cloud Functions, and Cloud Pub/Sub.
• Proficiency in Python, Spark, and SQL.
• Experience with container technologies such as Docker and GKE.
• Working knowledge of version control using GitHub and CI/CD tools like Jenkins or Concousse.
• Strong understanding of application architecture and cloud design considerations.
• Experience with SQL, NoSQL databases, and RDBMS.
• Knowledge of Linux OS, shell scripting, and administration.

Preferred Skills

• Experience with AWS or Azure.
• Knowledge of streaming tools like Kafka or Kinesis.
• Experience implementing Infrastructure as Code (IaC) tooling.

Key Skills

Education

BE/BTech/MCA