Description

As an AI Quality Infrastructure Engineer, you will build quality infrastructure and build quality pipelines with these to guarantee the reliability of our AI ecosystem. You won’t just monitor models; you will build the automated systems that make monitoring possible at scale. You will be responsible for engineering the "LLM-as-a-judge" services, custom observability frameworks, and the automated alerting logic that connects our AI agents to our production response teams. Your work will bridge the gap between AI research and production-grade reliability engineering.

What the Job Entails

• Build Automated Quality Tooling for AI: Build and maintain internal tools and services that automate the measurement of quality for the AI and AI agent development lifecycle. This includes the development of quality coverage tools for prompt-based approaches, creating testing automation pipelines to support tool call validations, and the "LLM-as-a-judge" scoring engines.
• Build Production Monitoring Tooling: Build and maintain internal tools the ensure continuous production monitoring with synthetic test for our AI capabilities.
• Design Synthetic Data Generators: Build tools to programmatically generate high-fidelity synthetic datasets for continuous stress-testing and "golden set" benchmarking.
• Labeling tools: Build and maintain rapid labeling pipelines for AI Agents.
• Engineer Observability Pipelines: Develop the backend data pipelines that stream model logs, tool-calling traces, and metadata into Splunk and Amplitude for real-time visualization.
• Automate Alerting & Incident Response: Write the logic and scripts to programmatically trigger PagerDuty incidents based on complex model performance thresholds and data quality anomalies.
• Develop Data Quality Services: Create automated services to detect data drift and non-natural language patterns (e.g., input feature distributions or sentiment shifts) before they impact the user.
• Scalable ML Tooling: Eventually design and build the infrastructure for end-to-end Machine Learning pipelines, focusing on automated training data validation and model-check gatekeeping.

Our Ideal Candidate

• Education: Bachelor’s or Master’s Degree in Computer Science, Software Engineering, or a related technical field.
• Engineering Proficiency: Expert-level Python and SQL skills with a focus on building reusable libraries, APIs, and automation scripts.
• Monitoring-as-Code: Experience in "Monitoring-as-Code," including programmatically configuring Splunk alerts, Amplitude, and PagerDuty services.
• AI/ML Infrastructure: Strong understanding of LLM architectures and the engineering challenges of testing non-deterministic systems.
• System Design Mindset: Ability to design scalable, fault-tolerant systems that can handle millions of AI conversation traces without latency.
• Problem Solving: A "builder" mentality—you see a manual process and your first instinct is to write code to automate it

Key Skills

Education

Any Graduate