Top Highlights
-
Big tool catalogues hurt retrieval: As the number of tools grew from 1,000 to 3,794, every method’s score fell by about a fifth—even though the right answers stayed the same.
-
Bigger models didn’t solve it: BM25 keyword search performed about as well as much larger embedding models, while running far faster and without a GPU.
-
Better tool descriptions made a difference: Searching full documentation improved results more than upgrading to a much larger model; split code-style names like
get_factorsso keyword search can match them. -
Practical takeaway: Keep a small toolset in the prompt; once it reaches hundreds, retrieve first. Start with BM25 on full documentation—but expect even the best setup to miss a needed tool in roughly a third of tasks.
The drawer gets crowded
An agent with hundreds of tools faces a simple problem: Which one fits this request? Loading every tool into the prompt works at small scale. But as the catalogue grows, the prompt can exceed the model’s limit or take much longer to process.
A benchmark test held 200 questions and their correct answers steady while the tool list grew from 1,000 to 3,794. Every retrieval method lost ground as extra tools crowded the list.
Keywords kept pace with embeddings
BM25 matches words in a request with words in tool names and descriptions. Embedding models instead compare meaning, so they can connect different phrases, such as “find factors” and “return divisors.”
In this test, BM25 and a 335-million-parameter embedding model scored nearly the same on the largest tool list. A model 24 times larger did not score higher. BM25 also searched in less than a millisecond per request, without a GPU. However, tool names that clearly describe their function may have favored keyword search. Different tool sets could produce different results.
Better descriptions may beat bigger models
The strongest practical improvement came from giving the retrievers full tool documentation instead of brief descriptions. BM25’s score rose from 0.494 to 0.533 on the largest list. That result suggests teams should improve tool names and documentation before investing in a larger model.
Still, no method solved the problem. Even the best setup found every needed tool in its top 10 only about 65% of the time. For a few dozen tools, listing them in the prompt remains simple. For hundreds, retrieval makes sense. Start with BM25, then test embeddings or a hybrid approach on your own catalogue.
Stay Ahead with the Latest Tech Trends
Learn how the Internet of Things (IoT) is transforming everyday life.
Discover archived knowledge and digital history on the Internet Archive.
AITechV1
