Summary Points
- Classic Monty Hall: Switch—your odds rise from 1/3 to 2/3 because the host knows where the car is and deliberately reveals a goat.
- Change the host’s rule, change the answer: If the host opens a random door or offers a swap only under certain conditions, switching may be no better—or may always lose.
- The tested AI models mostly kept saying “switch.” They often ignored the host’s rules, even when their own explanations correctly repeated them.
- Takeaway: A famous puzzle’s familiar answer isn’t proof of reasoning. Change one key rule and check whether the answer changes too.
A familiar answer can mislead
In the classic Monty Hall problem, switching gives you a two-thirds chance of winning. But that answer depends on the host’s rules. If the host opens a random door without knowing where the car is, switching offers no advantage. If the host offers a swap only when you picked the car, switching always loses.
The doors stay the same. The evidence changes.
Small models kept choosing “switch”
A test put four Qwen2.5 models, from 0.5 billion to 7 billion parameters, and Phi-3.5-mini through six versions of the puzzle. The models often chose “switch,” even when the host’s behavior made that choice wrong. On four such versions, their average confidence in the correct answer ranged from 0% to 13%. Random guessing would score 33%.
Their written explanations showed the same pattern. Some models repeated the host’s rule, then reasoned as if the classic rules still applied. So a familiar answer could survive even after the problem changed.
Test the rule, not just the answer
The study is small. It covers a handful of compact models and puzzle variants, not every AI system. A correct answer to one famous puzzle also cannot prove that a model lacks reasoning. Still, the results offer a useful warning: fluent explanations and high confidence do not guarantee that a model noticed a changed assumption.
For developers and users, the practical lesson is simple. Change one rule and check whether the answer changes too. That kind of test can reveal whether a model follows the problem or reaches for a memorized response.
Expand Your Tech Knowledge
Dive deeper into the world of Cryptocurrency and its impact on global finance.
Discover archived knowledge and digital history on the Internet Archive.
AITechV1
