Essential Insights
- Models are improving: Newer systems are approaching the threshold on open-ended tasks, though they’re not yet useful for METR’s research.
- Progress is hard to predict: These tasks are difficult to design, score, and compare over time.
- The measurements have limits: Many human completion times are estimates, and some tasks rely on only one evaluator.
- Promising, but no clear timeline: Better evaluations are a step forward, but it’s too soon to know when AI could automate research and development.
A broader test, not a victory lap
METR’s time-horizon graph estimates how long a task would take a person, then compares that with AI performance. Its latest update adds more open-ended tasks, including work that requires design and judgment. That matters because those skills could shape future AI research.
The signal looks encouraging. A year ago, every model scored clearly below the bar on these evaluation-design tasks. Now, some of the newest models, such as Opus 5.5, sit near or just above it. Still, they are not yet useful for METR’s research.
The new scores need careful reading
The graph cannot settle how quickly AI will improve. METR includes few open-ended tasks because they are difficult to design and score. And while the original graph used people’s actual completion times, the new report relies mostly on estimates. Some tasks could take hundreds of hours or require rare expertise, making direct measurement costly.
Scoring can also hinge on individual judgment. For one philosophical argument task, a single person rated most of the arguments. Other experts might reach different conclusions. As a result, today’s scores offer clues, not firm forecasts.
Promising progress, uncertain adoption
Even so, adding these tasks moves evaluation toward work that resembles real research. That is a useful step for anyone tracking whether AI could help automate parts of AI development.
But companies and researchers should avoid treating the graph as a timetable. Models appear to be improving, yet the limited evidence makes future progress hard to predict. Better tests and broader expert review will help show whether today’s promising results can translate into dependable work.
Stay Ahead with the Latest Tech Trends
Explore the future of technology with our detailed insights on Artificial Intelligence.
Explore past and present digital transformations on the Internet Archive.
AITechV1
