Description

You will design and execute testing strategies for a GenAI-based, multi-agent, cloud-native chatbot platform built on AWS. Your focus is validating LLM-driven functionality, ensuring data and prompt security, and verifying integrations and performance.

This role is remote.

Responsibilities

• Develop and execute test plans for chatbot logic, GenAI responses, and agent interactions.
• Validate RAG functionality, including data retrieval accuracy, response relevance, and hallucination detection.
• Test API integrations between the chatbot, AWS services, and enterprise systems.
• Implement automation for regression, performance, and cost-efficiency testing.
• Ensure LLM prompt security through guardrails and injection testing.

Required Skills

• 10+ years of QA engineering experience.
• Strong proficiency in Python.
• Hands-on experience with AWS cloud services.
• Expertise in GenAI and RAG (Retrieval-Augmented Generation) architectures.
• Experience testing frameworks like LangGraph or similar agentic tools.
• Ability to validate LLM metrics, including hallucination detection and response accuracy.
• Understanding of prompt injection security and guardrails.
• Experience with API testing and automation.

Preferred Skills

• Experience with Drupal-based web integrations.
• Knowledge of analytics validation for user engagement and model performance.

Key Skills

Education

Any Graduate