Description

You will lead end-to-end data engineering lifecycles and solution design within a Google Cloud environment.

Responsibilities

• Design and develop ETL/ELT frameworks utilizing BigQuery features like nested queries, clustering, and partitioning.
• Build and manage data pipelines using Google Cloud data products and Airflow DAGs.
• Own end-to-end solution design, including prototyping, usability testing, and data visualization.
• Manage the full data engineering lifecycle, including non-functional requirements and operations.
• Automate data loads from BigQuery via APIs or scripting languages.

Required Skills

• 12+ years of professional experience in data engineering and architecture.
• Expert-level Python programming with deep experience in PySpark and Pandas.
• Hands-on experience with Google Cloud Platform (GCP) and BigQuery (GBQ).
• Proficiency with Google data products: Dataflow, Dataproc, Dataprep, and Cloud Composer.
• Advanced Airflow skills, including creating DAGs, configuring variables, and scheduling.
• Deep knowledge of Big Data technologies: Spark, Hadoop, Hive, and MapReduce.
• Experience with SQL and NoSQL modern data stores.
• Proficiency in scripting and configuration using YAML and Python.
• Experience working in DevSecOps (CI/CD) environments.

Preferred Skills

• Experience using DBT to create lineage in GCP.

Key Skills

Education

Any Graduate