Description

You will build the automated quality infrastructure and pipelines that guarantee the reliability of the AI ecosystem. You engineer the systems that make monitoring possible at scale, bridging the gap between AI research and production-grade reliability.

Responsibilities

• Build and maintain "LLM-as-a-judge" services, custom observability frameworks, and automated alerting logic for AI agents.
• Develop automated quality tooling for prompt-based testing, tool call validation, and synthetic test generation.
• Design data quality services to detect data drift, non-natural language patterns, and sentiment shifts before they impact users.
• Engineer backend data pipelines that stream model logs and traces into Splunk and Amplitude for real-time visualization.
• Write logic to programmatically trigger PagerDuty incidents based on model performance thresholds and data anomalies.

Required Skills

• Expert-level Python and SQL for building reusable libraries, APIs, and automation scripts.
• Experience with "Monitoring-as-Code," including programmatically configuring Splunk alerts and PagerDuty services.
• Strong understanding of LLM architectures and engineering challenges of testing non-deterministic systems.
• Ability to design scalable, fault-tolerant systems handling millions of AI conversation traces.
• Proficiency in building rapid labeling pipelines for AI Agents.
• 5+ years of engineering experience.
• Bachelor’s or Master’s degree in Computer Science, Software Engineering, or a related technical field.

Preferred Skills

• Experience with Amplitude for real-time data visualization.

Key Skills

Education

Any Graduate