Description

You will build and maintain automated tests for AI agent workflows, APIs, and telemetry analysis to ensure release quality. This role is on-site.

Responsibilities

• Design and implement evaluation suites for LLM and agentic behavior, including rubric-based grading, regression datasets, and safety checks.
• Validate multi-turn scenarios, knowledge retrieval, escalation paths, and on-device agent behavior, including Windows service and telemetry collection.
• Debug quality issues by reading logs, tracing requests, and making targeted code or test changes without heavy engineering hand-holding.
• Partner with engineering and product to define release gates, quality metrics, and confidence thresholds for pilot readiness.
• Contribute to CI quality checks, test fixtures, mocked integrations, and regression suite maintenance.

Required Skills

• 4+ years of experience in QA engineering, SDET, test automation, or software engineering.
• Strong Python proficiency, including writing production-quality tests and debugging application code.
• Experience testing backend services, APIs, async workflows, and distributed systems.
• Hands-on experience with modern test frameworks such as pytest, unittest, Playwright, or xUnit.
• Ability to test ambiguous AI behavior using datasets, rubrics, assertions, and regression baselines.
• Strong written communication for documenting repro steps, risks, and test plans.
• Bachelor's degree in a relevant field.

Preferred Skills

• Direct experience with Pydantic, Pydantic AI, or Pydantic Evals.
• Experience testing LLM applications, RAG, tool calling, or MCP servers.
• Experience with Windows endpoint testing, PowerShell, .NET, MQTT, or IT support tooling.

Key Skills

Education

Bachelor's degree