Quick Takeaways
-
More models don’t automatically mean more independent opinions: correlated mistakes can sharply limit the gains from voting.
-
In a test of six open models on MMLU, average error correlation was 0.41—making six models worth just about two independent votes.
-
Voting didn’t beat the best individual model: five-model voting scored 71.2%, while the best single model scored 72.7%. Probability averaging helped slightly, but not significantly.
-
Different developers didn’t eliminate shared errors. Measure how often models fail together before adding voters—and note this study tests voting, not agent teams that split up tasks.
Your Six AI Models Are Really Two: The Math of Correlated Mistakes
A vote works only when opinions differ
Adding models can improve a vote — if their mistakes don’t overlap. The classic math assumes each voter gets the answer right independently. Under that assumption, more voters steadily improve the odds.
But AI models may share blind spots. When several fail on the same questions, they don’t provide several independent opinions. A simple estimate captures the effect: effective votes = model count ÷ [1 + (model count − 1) × error correlation]. As correlation rises, each added model contributes less.
Six models, about two votes
In a test of six open models, researchers compared answers to 1,500 multiple-choice questions. The models came from five developers and scored between 58.3% and 72.7% individually. Their average error correlation was 0.41. By the formula, six models supplied about 1.96 independent votes.
The results matched that estimate. A five-model majority scored 71.2%, while the strongest single model scored 72.7%. Averaging all six models’ answer probabilities reached 73.8%, but that gain was not statistically clear. The models often missed the same questions.
This test does not settle the value of AI agent teams. A separate coding study tested agents that divided tasks among themselves, not models voting on the same answer. Teams cost more, and only one of four comparisons showed a statistically significant improvement. Those findings address a different question.
Measure before you add
Correlated errors do not mean model panels have no value. A second model can catch a mistake, and probability averaging may help. Still, a different developer does not guarantee a different perspective. In this test, models from different developers made errors that remained closely related.
Before expanding a panel, test it on examples with known answers. Measure which mistakes the models share. Then compare the added accuracy with the added cost. If errors cluster, a new tool, a reliable test, or human review may add more value than another similar model.
Discover More Technology Insights
Stay informed on the revolutionary breakthroughs in Quantum Computing research.
Discover archived knowledge and digital history on the Internet Archive.
AITechV1
