Description

You will architect, implement, and manage complex data pipelines and analytics solutions on Google Cloud Platform.

Responsibilities

• Design and develop data pipelines using Dataproc Serverless and Apache Spark.
• Manage workflows with Google Cloud Composer and convert Oozie scheduler jobs to Composer-based jobs.
• Integrate BigQuery and Google Cloud Storage into existing data architectures.
• Optimize Spark jobs, BigQuery queries, and data pipelines for performance and cost-effectiveness.
• Perform data validation both pre- and post-pipeline migration.

Required Skills

• 10+ years of experience in data engineering.
• Expertise in Dataproc Serverless and Apache Spark.
• Hands-on experience with Google Cloud Composer and Oozie scheduler.
• Proficiency in BigQuery and Google Cloud Storage (GCS).
• Advanced SQL skills and experience with data modeling.
• Proven track record in cloud cost optimization.
• Ability to identify and resolve performance bottlenecks in data processing.

Preferred Skills

• Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.

Key Skills

Education

Bachelor's or Master's degree in Computer Science, Information Technology