Description

Own the product roadmap for LLM post-training and evaluation capabilities, including benchmark infrastructure and model quality platforms. Drive delivery from concept to production by partnering with Research Scientists and AI/ML Engineers.

Responsibilities

• Define and own the product roadmap for LLM evaluation, benchmark infrastructure, and post-training workflow tooling.
• Conduct deep discovery with AI labs and enterprise ML teams to identify evaluation pain points and synthesize structured requirements.
• Translate complex technical requirements into actionable product specifications and make rigorous prioritization decisions across the backlog.
• Partner with Sales, Solutions Engineering, and Marketing to develop positioning, packaging, and pricing for evaluation offerings.
• Define and track product KPIs for adoption and quality, using data to drive decisions and communicate progress to leadership.

Required Skills

• 5+ years of product management experience, with 3+ years specifically in AI/ML, data platform, or developer tools products.
• Deep understanding of LLM evaluation approaches: automated metrics, human evaluation, model-as-judge, and hybrid frameworks.
• Knowledge of post-training techniques including SFT, RLHF, RLAIF, DPO, PPO, and GRPO.
• Ability to reason about tradeoffs in benchmark design regarding validity, reproducibility, scalability, and cost.
• Experience productizing ML tooling, evaluation pipelines, or data workflows into repeatable, scalable platform capabilities.
• Proficiency in defining APIs, data contracts, and integration patterns in collaboration with engineering teams.
• Strong technical fluency to engage credibly with research scientists and ML Engineers on technical tradeoffs.

Preferred Skills

• Familiarity with long-context, multimodal, and agentic evaluation challenges.
• Experience supporting enterprise sales cycles with product expertise and technical storytelling.

Key Skills

Education

Any Gradute