Description

You will design and build realistic benchmark tasks for coding agents using production-like repositories, tests, and runtime scenarios.

This role is remote.

Responsibilities

• Create benign engineering tasks including bug fixes, feature additions, CI repairs, and config migrations.
• Define utility and safety/alignment requirements to verify agent correctness and constraint preservation.
• Develop visible and hidden test suites to evaluate task completion and identify unsafe agent behavior.
• Analyze rollouts from frontier coding agents such as Claude Code and Codex to assess utility and safety violations.
• Collaborate with engineering and security teams to improve task quality and evaluator reliability.

Required Skills

• 8+ years of hands-on software engineering experience in product companies or technology startups.
• Strong programming proficiency in Python (mandatory) and at least one of Java, Go, or C++.
• Bachelor's or Master's degree in Computer Science or a related technical discipline.
• Experience building and maintaining production software used by real customers.
• Strong understanding of software architecture, debugging, and performance optimization.
• Production engineering experience with deployment pipelines, monitoring, logging, and observability.
• Ability to diagnose production failures using logs, metrics, and distributed tracing.
• Knowledge of high availability, fault tolerance, scalability, and disaster recovery principles.
• Experience identifying security vulnerabilities through code reviews and implementing secure code fixes.

Preferred Skills

• Practical knowledge of common software vulnerabilities including injection attacks, memory safety issues, and deserialization vulnerabilities.
• Experience creating safe and unsafe reference solutions with scoring rubrics and calibration notes.

Key Skills

Education

Bachelor's or Master's degrees