Description

You will own the design and maintenance of evaluation pipelines for prompt and agent reliability, accuracy, and latency.

This role is hybrid.

Responsibilities

• Build prompt experimentation tools, including versioning, benchmarking, and result comparison for internal teams.
• Improve context management strategies, chunking, summarization, and vector-based retrieval processes.
• Contribute to dashboards and telemetry pipelines to track agent health, drift, and usage patterns.
• Collaborate with platform engineers to validate experiment outputs and improve system behavior.
• Document and advocate for best practices in LLM evaluation, experiment design, and prompt testing.

Required Skills

• 2–3 years of experience in data science or applied ML roles, ideally involving AI systems or infrastructure.
• Strong Python skills, including experience with APIs, data processing, and testing frameworks.
• Hands-on experience designing and running structured experiments, including metric design and result interpretation.
• Familiarity with LLM fundamentals, prompt engineering concepts, and evaluation techniques.
• Experience with Python, APIs, Data Processing, LLM, CI/CD, and .NET/C#.
• Solid communication skills, capable of clearly documenting findings and explaining tradeoffs.

Preferred Skills

• Experience with LLM evaluation libraries (e.g., TruLens, RAGAS) or test harnesses.
• Familiarity with vector stores and retrieval (e.g., FAISS, Weaviate, Qdrant).

Key Skills

Education

Any Graduate