Description

You will design and implement data pipelines within the Google Cloud Platform ecosystem.

Responsibilities

• Develop ETL processes for data ingestion, capture, processing, and curation.
• Build real-time streams to enable data collection, storage, and transformation.
• Leverage GCP components including Dataflow, Dataproc, and BigQuery for data engineering workflows.
• Automate deployments and testing through CI/CD pipelines.
• Schedule complex workflows using Airflow and Cloud Composer.

Required Skills

• 5+ years of experience in data engineering or ETL development.
• Hands-on experience with GCP services: BigQuery, Dataflow, Dataproc, Composer, Pub/Sub, and Cloud Storage.
• Proficiency in Python and Java.
• Experience with SQL and database technologies including BigQuery, BigTable, and Teradata.
• Strong command of Bash shell scripting, UNIX utilities, and UNIX commands.
• Experience with CI/CD automation pipelines.
• Ability to work with JSON and XML data formats.
• Experience using JIRA or similar project management tools.

Preferred Skills

• Experience with Spark, PySpark, or Kafka.
• Knowledge of Kubernetes and Docker containerization.
• Familiarity with Scrum/Agile methodologies.

Key Skills

Education

Any Gradute