Description

Key Skills: Application Security Testing, Retrieval-Augmented Generation (RAG), LLM

Roles and Responsibilities:

• Design and implement model evaluation frameworks using RAGAS, custom scorers, and automated accuracy benchmarking to validate agent outputs against structured specifications and Gherkin acceptance criteria.
• Develop adversarial prompt testing suites to probe agent behavior for hallucinations, prompt injection vulnerabilities, and edge-case failures across refinement, decision, and coding agents.
• Build and maintain regression test suites that continuously verify agent behavior consistency across model updates, prompt changes, and skill-file modifications within a LangGraph-based architecture.
• Integrate AI-specific test tooling into CI/CD pipelines using GitHub Actions, ensuring agent deployments are gated by automated quality, fairness, and reliability checks aligned with EU AI Act and DORA requirements.
• Define and track quantitative quality metrics (confidence thresholds, retrieval precision, response latency, drift detection) and deliver actionable dashboards to engineering teams.

Skills Required:

• 5 - 12 years of experience in testing AI/ML systems or LLM-based applications
• Application Security Testing
• Retrieval-Augmented Generation (RAG) testing and evaluation
• Automated accuracy benchmarking and regression testing for LLM/agent outputs
• CI/CD quality gating using GitHub Actions for AI agent deployments

Good to Have:

• LLM evaluation and quality assessment experience

Education: Bachelor's degree in Computer Science, Engineering, or a related field

Key Skills

Education

Any Graduate