Description

You will build automated quality infrastructure and pipelines to guarantee the reliability of the AI ecosystem. This role engineers the systems that enable monitoring at scale, bridging the gap between AI research and production-grade reliability.

This role is hybrid.

Responsibilities

• Build and maintain LLM-as-a-judge services, custom observability frameworks, and automated alerting logic for AI agents.
• Develop automated quality tooling for prompt-based testing, tool call validation, and synthetic test generation.
• Design data quality services to detect data drift, non-natural language patterns, and sentiment shifts before they impact users.
• Engineer backend data pipelines streaming model logs and traces into Splunk and Amplitude for real-time visualization.
• Write logic to programmatically trigger PagerDuty incidents based on model performance thresholds and data anomalies.

Required Skills

• Expert-level Python and SQL for building reusable libraries, APIs, and automation scripts.
• Experience with Monitoring-as-Code, including programmatically configuring Splunk alerts and PagerDuty services.
• Strong understanding of LLM architectures and engineering challenges of testing non-deterministic systems.
• Ability to design scalable, fault-tolerant systems handling millions of AI conversation traces.
• Proficiency in building rapid labeling pipelines for AI Agents.
• 5+ years of engineering experience.
• Bachelor’s or Master’s degree in Computer Science, Software Engineering, or a related technical field.

Preferred Skills

• Experience with Amplitude for real-time data visualization.

Key Skills

Education

Any Graduate