Description

You will build and manage data pipelines and infrastructure within the Google Cloud Platform ecosystem.

Responsibilities

• Develop and maintain data pipelines using Python, Scala, and PySpark.
• Manage data lake and data warehouse architectures using BigQuery and SQL.
• Implement data quality engineering and testing protocols.
• Orchestrate workflows using GCP components like Airflow (DAGs) and Dataflow.
• Manage data governance and metadata management processes.

Required Skills

• 11+ years of experience in data engineering.
• Google Cloud Platform (GCP) expertise.
• Proficiency in Python and PySpark.
• Experience with BigQuery, SQL, and data warehousing.
• Hands-on work with PubSub and gsutil commands.
• Knowledge of Airflow (DAGs) and Dataflow.
• Experience with REST API testing and API performance testing.
• Implementation of CI/CD via Azure DevOps.
• Background in data governance and metadata management.

Preferred Skills

• Experience with Scala in a data engineering context.

Key Skills

Education

Any Graduate