Description

Evaluate Large Language Models, generative AI applications, and conversational systems for accuracy, safety, and performance.

This role is on-site.

Responsibilities

• Conduct functional, regression, and UAT testing for AI-powered applications and chatbots.
• Design and execute test cases to evaluate AI responses for accuracy, relevance, consistency, and safety.
• Perform prompt engineering and variation testing to identify model limitations, edge cases, and hallucinations.
• Validate API responses and integrations using Postman and document defects in Jira.
• Annotate datasets using Label Studio and analyze testing results to generate quality reports in Excel and Google Sheets.

Required Skills

• 10+ years of experience in quality assurance, with specific focus on AI/LLM evaluation.
• Proficiency in evaluating generative AI models (ChatGPT, Claude, Gemini, Microsoft Copilot).
• Strong prompt engineering skills, including designing variations to stress-test models.
• Experience with bug tracking and project management tools, specifically Jira.
• API testing capabilities using Postman.
• Data annotation and review experience using Label Studio.
• Advanced data analysis and reporting skills using Microsoft Excel and Google Sheets.
• Understanding of AI safety, content guidelines, and bias identification.

Preferred Skills

• Experience with manual testing methodologies and functional testing frameworks.
• Background in collaborating with product and engineering teams to improve model performance.

Key Skills

Education

Any Graduate