Description

You own end-to-end quality assurance for AI-driven applications, ensuring reliability across UI, API, and probabilistic model outputs.

This role is on-site.

Responsibilities

• Build and maintain automation frameworks using Playwright, Cypress, or Selenium for UI and pytest/REST Assured for API testing.
• Design test strategies for non-deterministic AI outputs, implementing statistical assertions, golden datasets, and human-in-the-loop reviews.
• Integrate test suites into CI/CD pipelines via GitHub Actions, Jenkins, or GitLab CI, enforcing quality gates and reporting.
• Validate RAG systems and LLM behaviors, checking retrieval precision, citation validity, and safety compliance like hallucination detection.
• Manage test data lifecycle, ensuring secure handling of PII and creating edge-case datasets for structured and unstructured inputs.

Required Skills

• 5+ years of experience in QA automation, with strong proficiency in Python and Java.
• Hands-on expertise with Playwright, Cypress, or Selenium for web interface testing.
• Proficiency in API testing tools such as Postman, Newman, REST Assured, and pytest.
• Experience integrating tests into CI/CD environments using Git, GitHub Actions, Jenkins, or GitLab CI.
• Ability to handle non-deterministic testing through variance thresholds, repeated runs, and prompt versioning.
• Understanding of observability tools (OpenTelemetry, ELK, Datadog) for tracing AI system behavior.
• Knowledge of cloud platforms (AWS/Azure/GCP) and containerization (Docker).

Preferred Skills

• Familiarity with NLP concepts, embeddings, and AI frameworks like LangChain or LlamaIndex.
• Experience with performance testing for inference endpoints, including latency, throughput, and cost-aware metrics.

Key Skills

Education

Bachelor's degree