{"id":23859,"date":"2026-09-09T23:17:25","date_gmt":"2026-09-09T23:17:25","guid":{"rendered":"https:\/\/scannn.com\/hyper-%f0%9d%9c%8f-bench-evaluating-agents-that-build-agents\/"},"modified":"2026-09-09T23:17:25","modified_gmt":"2026-09-09T23:17:25","slug":"hyper-%f0%9d%9c%8f-bench-evaluating-agents-that-build-agents","status":"publish","type":"post","link":"https:\/\/scannn.com\/lv\/hyper-%f0%9d%9c%8f-bench-evaluating-agents-that-build-agents\/","title":{"rendered":"Hyper-\ud835\udf0f-bench: Evaluating agents that build agents"},"content":{"rendered":"\n
\n
\n

We built \ud835\udf0f-bench in 2024<\/a> to answer a question that felt novel at the time: Can a model act as a reliable customer service agent? That\u2019s table stakes now. The harder question is, who\u2019s building the agent in the first place? Increasingly, it\u2019s the models themselves.<\/p>\n

We\u2019ve partnered closely with some of the world\u2019s leading companies to launch their agents. In practice, the work is less like implementing a spec, and more like doing research. Requirements are scattered across handbooks, support, spreadsheets, and the minds of your best frontline reps \u2014 so you form a hypothesis, dig up evidence, and build and test to identify which levers actually move performance.<\/p>\n

Today we\u2019re open-sourcing hyper-\ud835\udf0f-bench (published as \ud835\udf0f^\ud835\udf0f-bench), a new long horizon agent evaluation that measures how well models can not only act as an agent, but construct one.<\/p>\n<\/div>\n

\n
The hyper-\ud835\udf0f-bench setup: A developer agent recovers requirements from a business’s records, builds a customer-service agent in a sandboxed workspace, and is scored on how that agent handles held-out tasks.<\/figcaption><\/figure>\n<\/div>\n
\n

Inside the sandbox<\/strong><\/h2>\n

Hyper-\ud835\udf0f-bench drops a developer agent into a sandboxed workspace with the records of a simulated business, plus a simulated client that it can message at any time. From there, the developer agent does the job end-to-end \u2014 it recovers the spec from the evidence, designs the architecture, and turns the business\u2019s actions into tools \u2014 until it has a working customer-service agent. The client\u2019s REST API may be subtly defective, so part of the job is figuring out whether a bug is in the spec or in the code. The finished agent has to serve from a fixed menu of models, within a cost budget per conversation. Once it\u2019s handed off, we deploy it against simulated production traffic using fully verifiable \ud835\udf0f-bench-style tests the developer never saw while building.<\/p>\n

Where the frontier stands today<\/strong><\/h2>\n

Working alone, our best configuration \u2014 Claude Opus 5 (max reasoning) running in Claude Code \u2014 passes just 23.9% of the held-out evaluation tasks. Paired with an engineer with deep context, the same class of model reaches 82.2% on the same tasks.<\/p>\n<\/div>\n

\n
Pass rate vs. human + model reference, plus build time and spend.<\/figcaption><\/figure>\n<\/div>\n
\n
Architecture, model choices, and rate of cheating-adjacent attempts.<\/figcaption><\/figure>\n<\/div>\n
\n

We read through developer trajectories to see where their builds lost ground. Five patterns stood out:<\/p>\n