Description

You will design and maintain data ingestion, processing, and transformation pipelines feeding AI knowledge bases on GCP.

Responsibilities

• Design and implement data ingestion pipelines from GCS, BigQuery, and Google Drive.
• Build workflows for knowledge base indexing and vectorization (RAG pipelines).
• Implement incremental updates, targeted revisions, and full rebuild options for AI agents.
• Collaborate with ML Engineers to integrate data pipelines with Vertex AI models.
• Optimize pipelines for cost, performance, and scalability within GCP.

Required Skills

• 4+ years of Data engineering experience.
• Strong programming skills in Python and SQL.
• Proficiency with GCP data services: BigQuery, Cloud Storage, Dataflow, Pub/Sub, Composer/Airflow.
• Experience with ETL/ELT pipeline design and workflow orchestration.
• Familiarity with vector databases/embeddings (e.g., Pinecone, FAISS, Vertex AI Matching Engine).
• Knowledge of data modeling, governance, and version control.
• Experience with CI/CD, Docker, and Kubernetes for data pipeline deployment.
• Strong analytical and debugging skills.

Key Skills

Education

Any Graduate