Description

You will support and scale the SAKS agentic AI platform, focusing on evaluation frameworks, RAG optimization, and operational excellence across cost, latency, and reliability.

This role is on-site.

Responsibilities

• Build and maintain evaluation suites for agents, including test datasets, scoring logic, regression tests, and guardrails.
• Configure, optimize, and monitor LLM integrations (Bedrock, Claude, OpenAI, Azure OpenAI) and RAG pipelines.
• Implement experiment tracking, CI/CD pipelines, and automated deployment with rollback capabilities.
• Set up observability for logs, metrics, and traces, defining alerts and handling incident response and RCA.
• Monitor and optimize token usage, latency, throughput, and infrastructure costs.

Required Skills

• 5+ years of experience with Python, specifically in LLM and Agent frameworks (LangChain/LangGraph).
• Hands-on expertise with vector databases (pgvector, Pinecone, Weaviate) and RAG pipeline components (embeddings, retrieval, chunking, prompt tuning).
• Strong experience with experiment tracking and evaluation frameworks (MLflow or equivalent).
• Proficiency with cloud platforms (AWS, Azure, GCP) and containerization (Docker).
• Experience with CI/CD, Git, and deployment practices.
• Knowledge of monitoring, logging, and cost optimization strategies.
• Familiarity with FastAPI or similar frameworks for service exposure.
• Understanding of structured outputs and schema validation (JSON/Pydantic).

Preferred Skills

• Experience with RAG evaluation frameworks (RAGAS, OpenAI Evals) and multimodal pipelines.
• Exposure to CrewAI, AutoGen, LlamaIndex, Airflow/Dagster, or PySpark.
• Knowledge of MLOps tools, model registry, Kubernetes, Terraform, and security/PII handling.

Key Skills

Education

Any Graduate