Description

 Design and build realistic coding-agent benchmark tasks using production-like repositories, tests, configs, documentation, and runtime scenarios.

Create benign engineering tasks such as bug fixes, feature additions, CI repairs, config migrations, integration updates, and runtime-state fixes.

Define clear utility requirements to verify that the coding agent completes the requested task correctly.

Define safety/alignment requirements to ensure the agent preserves system constraints, developer intent, data integrity, privacy, permissions, and oversight mechanisms.

Develop visible and hidden test suites to evaluate both task completion and unsafe agent behavior.

Create safe reference solutions and unsafe reference solutions where the agent passes utility checks but violates multiple safety/alignment requirements.

Identify and evaluate unsafe shortcuts such as disabling tests, weakening assertions, deleting protected data, leaking secrets, broadening permissions, or bypassing validation.

Package tasks with prompts, metadata, runnable repos, evaluators, reference patches, scoring rubrics, and calibration notes.

Analyze rollouts from frontier coding agents such as Claude Code and Codex to assess utility completion and safety/alignment violations.

Collaborate with engineering, QA, security, and client teams to improve task quality, evaluator reliability, and benchmark difficulty.

Required Qualifications

Bachelor's or Master's degree in Computer Science or a related technical discipline.

Minimum 8 years of hands-on software engineering experience in leading product companies or technology startups.

Strong programming experience in Python (mandatory) along with one or more programming languages such as Java, Go, C++

Demonstrated experience building and maintaining production software used by real customers.

Strong understanding of software architecture, debugging, and performance optimization.

Required Technical Experience

Production engineering

Experience developing software deployed in production at scale.

Strong understanding of software deployment pipelines, monitoring, logging, and observability.

Experience diagnosing production failures using logs, metrics, and distributed tracing.

Knowledge of high availability, fault tolerance, scalability, and disaster recovery principles.

Secure Software Development

Experience identifying security vulnerabilities through code reviews or security assessments.

Practical knowledge of common software vulnerabilities, including:

Injection attacks

Authentication and authorization flaws

Memory safety issues

Race conditions

Deserialization vulnerabilities

Secrets management issues

Input validation failures

Experience implementing secure code fixes and validating remediation

Key Skills

Education

Bachelor's or Master's degrees