Description

You will design and manage end-to-end data engineering lifecycles and solution architectures.

Responsibilities

• Design and develop ETL/ELT frameworks using BigQuery, incorporating nested queries, clustering, and partitioning.
• Build and manage data pipelines using Google Cloud Platform datasets and Google data products.
• Own end-to-end solution design, including prototyping, usability testing, and data visualization.
• Manage data integration, transformation, quality, and lineage processes.

Required Skills

• 5+ years of professional experience in architecture and data engineering.
• Deep expertise in Google Cloud Platform (GCP) and BigQuery.
• Hands-on experience with Dataflow, Dataproc, Dataprep, and Cloud Composer.
• Advanced Python programming skills with expertise in PySpark and Pandas.
• Proficiency in Airflow, including creating DAGs, configuring variables, and scheduling.
• Strong background in Big Data technologies including Spark, Hadoop, Hive, and MapReduce.
• Experience with DBT for creating lineage in GCP.
• Proficiency in SQL, NoSQL modern data stores, and YAML.

Preferred Skills

• Experience working in a DevSecOps (CI/CD) environment.

Key Skills

Education

Any Graduate