Quick Takeaways
-
Small collections looked resilient: Across 298 questions and up to 1,801 research papers, search quality barely dipped—though the test may have been unusually easy.
-
Scale changed the picture: On 2.68 million Wikipedia passages, BM25’s success fell as collections grew, dropping about 40 points at full size.
-
Rank matters for answers: When relevant passages were pushed down or out of the top results, the answer model performed worse; putting the answer passage first largely closed the gap.
-
Test your real setup: Use questions from people who haven’t seen your documents, and track whether useful passages reach the top of the context—not just whether they’re somewhere in the collection.
Small collections stayed strong
BM25, a common text-search method, helps RAG systems find passages before an AI model answers. In a test of 298 questions across collections of up to 1,801 research papers, the strongest results barely shifted as the collections grew. The largest measured drop was 3.4 percentage points, below the study’s five-point threshold for a meaningful decline.
That is encouraging, but not a guarantee. The test used questions tied to specific papers, which may share words with those papers. It also covered far fewer documents than many real-world search systems.
Scale changed the picture
A larger test used 3,412 public questions and 2.68 million Wikipedia passages. BM25’s success rate at placing a judged, relevant passage in its top 10 fell as the collection grew. The first collection size to meet the study’s decline threshold was 100,000 passages, where the score dropped 11.1 points from the 10,000-passage baseline. At full size, the score fell by about 40 points.
A controlled test offered another clue. Adding 20 unrelated passages from the same Wikipedia page lowered the search score by 8.3 points on average. Often, those passages ranked highly because they repeated the page title—not because they answered the question. The test therefore flags a real risk, but it does not prove that semantically similar passages cause the same harm.
Test the system you plan to use
The results support a practical lesson: do not assume strong performance on a small collection will hold at scale. Test with questions from real users, and track whether useful passages reach the top of the context an AI model receives.
Still, the large test has limits. Its relevance labels did not mark every passage that might contain an answer, and the experiments tested only a narrow set of search setups. BM25 remains a useful, accessible baseline. Teams should measure it against their own documents and questions before choosing a retrieval method—or trusting it to carry a larger RAG system.
Continue Your Tech Journey
Stay informed on the revolutionary breakthroughs in Quantum Computing research.
Access comprehensive resources on technology by visiting Wikipedia.
AITechV1
