Description

You will design and manage end-to-end data engineering lifecycles and pipelines on Google Cloud Platform.

Responsibilities

• Design and develop ETL/ELT frameworks using BigQuery, implementing clustering, partitioning, and nested queries.
• Build and manage data pipelines using Google Data Products including Dataflow, Dataproc, Dataprep, and Cloud Composer.
• Automate data loads from BigQuery using APIs or scripting languages.
• Own the end-to-end solution design process, including prototyping, usability testing, and data visualization.
• Manage data integration, transformation, quality, and lineage across the data lifecycle.

Required Skills

• 10+ years of professional experience in Data Engineering.
• Expert-level Python programming including PySpark and Pandas.
• Deep experience with Google Cloud Platform (GCP) and BigQuery (GBQ).
• Hands-on experience with Airflow (creating DAGs, configuring variables, and scheduling).
• Proficiency in Big Data technologies including Spark, Hadoop, Hive, and MapReduce.
• Strong SQL and NoSQL skills.
• Experience with scripting languages including YAML.
• Background working in DevSecOps (CI/CD) environments.

Preferred Skills

• Experience using dbt to create lineage in GCP.

Key Skills

Education

Any Graduate