Description

• Design, develop, and deploy AI agents that automate engineering workflows — translating requirements into working agentic systems that configure experiments, analyze results, identify regressions, and recommend follow-up actions.
• Build and maintain the evaluation harness for agentic systems: regression suites, golden sets, and scoring rubrics for non-deterministic outputs, integrated into the release pipeline so changes ship on evidence, not intuition.
• Architect the ingestion and storage pipeline for agent telemetry at scale, keeping the query surface fast and actionable during live incidents.
• Instrument cost, latency, and token consumption per agent, per tool, and per routing path, surfacing efficiency signals to engineers and architects shaping platform decisions.
• Advance the trace and event model for agent execution — tool calls, routing decisions, handoffs, retries, terminal outcomes — so any production run can be reconstructed and explained.
• Collaborate with data science partners on anomaly detection, drift detection, and failure clustering across agent execution history.
• Deliver instrumentation as composable libraries, making observability standards the path of least resistance for agent authors across the organization.
• Lead design reviews and shape observability and evaluation best practices across the agentic ecosystem

Key Skills

Education

Any Gradute