BonrockWhat an AI Time-Horizon Score Leaves Out
A task-completion horizon combines task difficulty with a chosen success rate. Reading the definition, scoring rules and costs gives a more useful view of AI progress.
METR evaluated the same model and arrived at a time horizon of about 11.3 hours or more than 270 hours, depending on how it counted attempts to cheat. Those numbers appeared in its June 2026 evaluation of GPT-5.6 Sol. METR did not consider either a robust capability measurement.
That is a difficult result to compress into a chart. A much larger number looks like progress until you inspect what was counted as success.
For someone deciding whether to buy or deploy an agent, the scoring rule is part of the finding. A model that reaches the expected answer by obtaining hidden test information has demonstrated a different capability from one that solves the assigned problem under the stated constraints. The distinction can disappear when a number travels from a technical report into a launch summary and then a budget presentation.
“Hours” needs a success rate attached
METR's time horizon uses human task duration as a measure of difficulty. The 50% horizon is where a fitted curve predicts success half the time on its task distribution. It does not measure how long the agent runs. The tasks mainly cover software, machine learning and cybersecurity, with clear scoring criteria.
That definition leaves several questions open for a buyer. How often does this particular job need to succeed? Can a person recognize a failed attempt? What happens before anyone notices the failure?
A simple calculation shows why the reliability question matters. Suppose a workflow requires five steps, each with a 50% chance of success, and suppose the outcomes are independent. The chance that all five succeed is 0.5 to the fifth power, about 3%. This hypothetical calculation assumes independent steps and no recovery; a real workflow needs evidence about both.
A half-success rate might be useful when generating candidate solutions that a cheap test can reject. It could be intolerable for an unattended change to a customer account. The same capability result can justify trying the first use and postponing the second.
The public page now says it is no longer actively updated and warns that measurements above sixteen hours are unreliable with its suite. That notice establishes a measurement limit; METR has not explained why maintenance stopped.
A trend can change the bill before it changes the job
Price is easier to compare when the quality requirement stays fixed. Epoch AI's study of inference prices measured the falling cost of reaching benchmark performance milestones. Its estimated annual declines ranged from 9-fold to 900-fold across the milestones studied, with caution about whether the fastest declines would persist.
A procurement decision still needs a unit of useful output. A low token price can be overwhelmed by repeated attempts, longer reasoning runs, or a reviewer who spends ten minutes finding the mistake in a two-minute draft. Conversely, a more expensive model may cost less per accepted result if it needs fewer corrections.
To examine a new release, we would return to the reason an earlier candidate was rejected. If it produced accurate work but cost too much, a price cut is directly relevant. If it mishandled exceptions, cheaper tokens leave that objection unresolved. If the output was acceptable but staff could not use it in their system, a connector or an interface change may matter more than a benchmark gain.
A release can change one of these decisions before the team has settled any forecast about general intelligence.
Even the productivity evidence changes when people opt out
Measuring the effect on real work introduces another problem. In February 2026, METR reported difficulties with its follow-up developer productivity experiment: some developers did not want to participate if they might have to work without AI. Selection and timing problems made the current speedup hard to estimate. The researchers thought speedups had likely improved, while calling their data very weak evidence for the size of the increase.
The refusal itself is informative about what those developers preferred. It cannot tell us how much time AI saved them.
Our preference at Bonrock is to record the condition that would change a deployment decision. For a coding assistant, that might be fewer failed builds or less time correcting patches. For a document tool, it might be a lower cost per accepted extraction with the same error tolerance. The condition should be chosen before seeing the next launch score; otherwise, it is easy to select whichever metric makes the purchase look sensible.
A small trial can uncover a reason to reject a tool. It will usually need more evidence to establish reliability, especially for rare and costly errors.
When the next time-horizon chart arrives, read the reliability level and scoring rules before comparing the line with last month's. Then return to the unresolved objection in your own deployment decision. Perhaps the release answers it. If it does, there is something to change; if it does not, a rising line can remain interesting without becoming a purchase order.
Sources
- METR, Summary of METR's predeployment evaluation of GPT-5.6 Sol, June 2026
- METR, Task-Completion Time Horizons of Frontier AI Models, page last updated September 2026
- Ben Cottier, Ben Snodin, David Owen and Tom Adamczewski, LLM inference prices have fallen rapidly but unequally across tasks, Epoch AI, March 2025
- METR, We are Changing our Developer Productivity Experiment Design, February 2026
Found this helpful? Pass it on.
Working on something like this?
We help organizations build AI systems that are secure, private and ready for real load.