Description

Design test strategies for AI applications including LLMs, RAG systems, conversational AI, and agentic workflows. Build automated evaluation pipelines for accuracy, groundedness, context relevance, reasoning, tool-use correctness, and latency. Create and maintain golden datasets, test scenarios, and evaluation metrics for regression testing. Implement LLM-as-a-Judge, schema validation, tracing, and calibrated scoring for multi-turn and long-horizon tasks. Test for hallucinations, bias, harmful content, policy violations, PII leaks, and brand drift using filters and guardrails. Conduct red-team testing for prompt injection, jailbreaking, data exfiltration, and adversarial inputs. Embed evaluation harnesses into CI/CD pipelines and monitor production telemetry for behavioral drift and failures. Analyze model and integration failures to identify root causes, controls, and mitigations. Translate business risks into measurable quality criteria, thresholds, and release recommendations. Collaborate with engineers, product managers, and security teams on testability, observability, and governance.

Skill and Ability:

3+ years of experience in QA engineering, test automation, ML QA, AI evaluation, or MLOps. Strong proficiency in Python and test frameworks such as Pytest, with experience building automated test suites. Hands-on experience testing LLM applications, RAG pipelines, or AI agents in production or pre-production environments. Understanding of AI failure modes: hallucinations, grounding errors, prompt sensitivity, context limits, and tool misuse. Familiarity with AI evaluation tools and platforms such as MLflow, Galileo, PyRIT, LangSmith, or equivalent. Knowledge of security testing concepts including prompt injection, jailbreaking, and data leakage risks. Experience working with CI/CD, containers, cloud environments, and observability tools. Strong analytical skills to interpret evaluation results and communicate risks clearly. Experience with multi-agent systems, autonomous workflows, or long-horizon task evaluation. Knowledge of responsible AI, fairness testing, and regulatory compliance for AI systems. Background in software development, data engineering, or ML engineering. Contributions to AI testing frameworks, open-source evaluation tools, or internal QA standards

Key Skills

Education

Bachelor's degree