Description

Own the quality and reliability of LLM-based agents and RAG applications through automated testing and evaluation frameworks.

This role is on-site.

Responsibilities

• Design and implement model evaluation frameworks using RAGAS and custom scorers to validate agent outputs against structured specifications and Gherkin acceptance criteria.
• Develop adversarial prompt testing suites to probe agent behavior for hallucinations, prompt injection vulnerabilities, and edge-case failures across refinement, decision, and coding agents.
• Build and maintain regression test suites that verify agent behavior consistency across model updates, prompt changes, and skill-file modifications within a LangGraph-based architecture.
• Integrate AI-specific test tooling into CI/CD pipelines using GitHub Actions, ensuring deployments are gated by automated quality, fairness, and reliability checks aligned with EU AI Act and DORA requirements.
• Define and track quantitative quality metrics (confidence thresholds, retrieval precision, response latency, drift detection) and deliver actionable dashboards to engineering teams.

Required Skills

• 5+ years of experience in testing AI/ML systems or LLM-based applications.
• Hands-on experience with Retrieval-Augmented Generation (RAG) testing and evaluation.
• Proficiency in Application Security Testing.
• Experience with automated accuracy benchmarking and regression testing for LLM/agent outputs.
• Ability to implement CI/CD quality gating using GitHub Actions for AI agent deployments.
• Bachelor's degree in Computer Science, Engineering, or a related field.

Key Skills

Education

Any Graduate