Description

Design and implement data processing frameworks and pipelines within Google Cloud Platform and Teradata environments, owning the lifecycle of batch and real-time data streams from ingestion to aggregation.

Responsibilities

• Build batch and real-time data pipelines for collection, storage, transformation, and aggregation.
• Develop ETL processes using SQL to analyze large datasets and generate structured outputs.
• Leverage GCP services to manage data ingestion, processing, and curation.
• Schedule complex workflows using Airflow or Cloud Composer.
• Automate deployments and testing through CI/CD pipelines.

Required Skills

• 12+ years of experience in data engineering or related roles.
• Strong SQL background for large-scale data analysis.
• Hands-on experience with GCP services: BigQuery, Dataflow, Cloud Storage, Dataproc, and Pub/Sub.
• Proficiency in Python and Java.
• Experience with Teradata and BigQuery/BigTable.
• Expertise in Airflow and Cloud Composer for workflow orchestration.
• Working knowledge of Bash shell scripting and UNIX utilities.
• Practical experience with CI/CD automation.

Preferred Skills

• Experience with Spark, PySpark, or Kafka.
• Knowledge of Kubernetes, Docker, and containerized GCP deployments.

Key Skills

Education

Any Graduate