I Tried Making my Own (Bad) LLM Benchmark to Cheat in Escape Rooms

Just lately, DeepSeek introduced their newest mannequin, R1, and article after article got here out praising its efficiency relative to value, and the way the discharge of such open-source fashions might genuinely change the course of LLMs eternally. That’s actually thrilling! And likewise, too massive of a scope to put in writing about… however when a mannequin like DeepSeek comes out of nowhere with a metal chair, boasting comparable efficiency ranges to different fashions, what does efficiency actually imply on this context?

For those who observe AI releases, you’ve seen this dance earlier than. Each new mannequin drops with its graphs displaying the way it’s one way or the other concurrently higher than GPT-4 on math issues whereas being smaller and extra environment friendly. However what precisely are these benchmarks measuring? How are they created? And extra importantly, how can we lower by means of the hype to create our personal benchmarks for particular use instances?

I wished to study extra about LLM Benchmarking.

Half 1: What’s a Benchmark? (in 3 seconds)

TL:DR — The SATs (a number of, truly) for LLMs.

Half 1.1: What’s a Benchmark? (in additional than 3 seconds)

Earlier than we dive into the nitty-gritty of particular benchmarks, let’s take a second to unpack what we even imply by “LLM Benchmark.” As a result of calling them the “SATs for AI” feels each proper and likewise barely oversimplified.

LLM benchmarks are, at their core, structured exams used to measure how properly giant language fashions carry out on sure duties. These duties could be something from figuring out if a press release is true or false, to summarizing a authorized doc, to producing legitimate Python features. Consider them as curated impediment programs specifically designed by AI researchers to check each related muscle these fashions may need. These frameworks usually present a dataset of inputs with recognized appropriate outputs, permitting for constant comparability between fashions.

Fashionable benchmarks make use of varied analysis methodologies. Classification metrics like accuracy work for duties with discrete appropriate solutions, whereas overlap-based metrics (BLEU, ROUGE) consider free-form textual content technology. Some benchmarks use useful testing for code technology, or make use of different LLMs as judges to guage response high quality.

A typical benchmark often comes packaged as:

• A standardized dataset of questions, prompts, or duties (with appropriate or reference solutions).
• An analysis protocol specifying the best way to measure success, like accuracy, F1 rating, BLEU/ROUGE for textual content technology, or go/fail charges for coding duties.
• A leaderboard or some type of comparative scoreboard, usually with massive flashy graphs.

Some actually well-known benchmarks embrace MMLU for testing multitask language understanding, TruthfulQA for assessing factual accuracy, and HumanEval for measuring coding capabilities. Outcomes are fairly usually printed on public leaderboards, which let’s individuals carry out some clear comparability between completely different fashions.

What Makes a Good Benchmark?

1. A Clear Job Definition: We would like duties which might be unambiguous. The extra easy and well-specified the problem, the simpler it’s to belief the outcomes.
2. Information Integrity: The check set shouldn’t be floating round within the coaching knowledge. As a result of if the mannequin’s seen the very same query 50 instances earlier than, the analysis is about as helpful as giving a math quiz to somebody who already has the reply key.
3. Quantifiable Metrics: You want a regular for scoring efficiency — like what number of instances the mannequin’s code passes check instances or how shut the generated abstract is to a “ground-truth” abstract.
4. Job Range & Issue: If a benchmark is just too simple, everybody simply ACES it on day one, and we study… properly, nothing. If it’s too area of interest (like “We check solely the mannequin’s capability to rely the digits of Pi for 20 minutes”), that’s additionally not so useful.

Life Ain’t All about The Grades

Benchmarks seize solely a slice of what LLMs can do. In the true world, your chatbot would possibly have to juggle area information, preserve monitor of dialog context, abide by your organization’s insurance policies, and produce fluent, non-offensive replies. No single standardized check on the market absolutely covers that. As we’ll see within the upcoming case research, the design and execution of a benchmark can closely form the image you get of your mannequin’s efficiency… and typically lead you astray should you’re not cautious with the way you measure success.

Now that we’ve a way of what Llm Benchmarks are designed to perform (and the place they could fall brief), let’s discover a few examples to see how individuals truly construct and use them in follow — with blended outcomes!

Case Examine #1: Leetcode as an LLM Benchmark

As a pupil within the tech house, the phrase “Leetcode” popping up throughout my seek for cool benchmarks raised by blood stress by a statistically vital quantity. In contrast to Leetcode, which sucks, the paper “Efficiency Examine of LLM-Generated Code on Leetcode” was very attention-grabbing — it asks a deceptively easy query: can we use Leetcode to benchmark LLM code technology? Their findings reveal each the promise and pitfalls of this method.

The Benchmark Design

The researchers constructed a three-stage validation system. Native exams catch primary errors, Leetcode’s decide verifies correctness, and a customized benchmarking setup measures efficiency. This setup revealed one thing important: benchmarking code efficiency is more durable than it seems to be.

After they in contrast native measurements to Leetcode’s metrics, they discovered solely a 0.28 correlation. Leetcode’s measurements confirmed a lot increased variation (0.089 vs 0.035 domestically). Even worse, Leetcode’s rankings proved unstable — equivalent options might drop from the 77th to 54th percentile simply based mostly on submission timing.

The Actual Issues

Three main points emerged that problem Leetcode’s viability as a benchmark:

Information Contamination: Utilizing public issues dangers LLMs having seen the options throughout coaching. The researchers had to make use of solely issues from 2023 to mitigate this.

Platform Instability: Leetcode’s metrics drift over time — reminiscence measurements confirmed a -0.24 correlation with check date. This makes reproducible benchmarking practically unattainable.

Measurement Reliability: The weak correlation between native and platform measurements raises questions on what we’re truly testing.

What It Means for LLM Benchmarking

This examine doesn’t simply critique Leetcode — it highlights what we’d like in a code technology benchmark: reproducible measurements, dependable efficiency metrics, and assured training-test separation. Till we’ve platforms constructed particularly for this goal, we should be extraordinarily cautious about utilizing competitors platforms as benchmarks.

So! We all know that not all benchmarks are viable benchmarks — what a few extra mainstream one?

Case Examine #2: SuperGLUE — Constructing a Higher Language Understanding Benchmark

The SuperGLUE paper tackles an enchanting drawback in AI benchmarking: what do you do when fashions get too good at your exams? When GLUE turned inadequate (with fashions surpassing human efficiency), the researchers needed to rethink how we measure language understanding.

The Benchmark Design

SuperGLUE’s core innovation is its activity choice methodology. The researchers collected activity proposals from the NLP group and filtered them by means of a rigorous course of: every activity wanted clear analysis metrics, public coaching knowledge, and — most significantly — vital headroom between machine and human efficiency.

This resulted in eight duties (I’ve simplified the desk from the doc right here, it’s rather less readable however it’s best to get the sense of what the questions are asking):

What makes these duties particular is their variety in format. In contrast to GLUE’s give attention to sentence classification, SuperGLUE contains coreference decision, studying comprehension, and extra com plex reasoning duties. Every activity measures completely different facets of language understanding whereas sustaining clear, quantifiable metrics.

---

Half 2: Let’s Construct a Bodily Reasoning Benchmark: To Cheat at Escape Rooms

After some benchmarks like SuperGLUE and Leetcode, I had an thought: what if we examined LLMs on one thing utterly completely different — bodily reasoning… by means of escape room puzzles?

It’s a fairly legitimate thought — escape rooms poses potentialities and penalties for failure — screw up one too many puzzles, and your folks will suppose you’re fairly silly, and relegate you to spectator obligation. Fortunately for us nonetheless, they (or the poor staff) don’t know that you could sneak a cellphone into an escape room — and you already know simply who to ask for the solutions. As we speak, LLMs face off towards the puzzles of a bodily escape room.

Word: That is NOT a rigorous tutorial benchmark (please don’t cite this in papers, why would you even need to do this?), and even near it, and it’s simply imagined to be a enjoyable method to check LLM benchmarking and analysis. Please don’t destroy my prompts, I’m conscious they’re dangerous.

Why Bodily Reasoning?

For actual, although… most LLM benchmarks give attention to linguistic duties (like SuperGLUE) or code technology (like Leetcode). And for good purpose — these are well-defined domains with clear analysis metrics. However real-world drawback fixing usually requires understanding bodily rules and their interactions. The well-known “Can GPT-4 do physics?” debates often focus on mathematical problem-solving, not sensible bodily reasoning.

Taking a look at present benchmarks taught me a number of key rules:

1. Clear analysis metrics are essential (from SuperGLUE’s task-specific scores)
2. Issues ought to have unambiguous options (from HumanEval’s check instances)
3. The benchmark ought to check distinct capabilities (from MMLU’s topic classes)

Designing the Issues

I settled on escape room puzzles for 2 causes. First, they naturally mix bodily reasoning with clear targets. Second, they’ve unambiguous success circumstances — both you clear up it by means of the meant approach, otherwise you don’t. Third, and most significantly, they let me embrace “pink herrings” — irrelevant gadgets that check if the LLM can determine what issues bodily. Fourth, I simply actually like doing escape rooms (did I point out that already?),

I’m conscious that that is greater than two causes, but when LLMs can’t rely what number of rs’ there are in strawberry, I’m allowed to mess up on occasion too.

Right here’s how I structured the 5 core issues:

Fluid Dynamics (FLUID_001) (Ping pong ball caught in a tube)

• Exams understanding of buoyancy and fluid displacement
• Impressed by basic physics issues however in sensible context
• Contains deliberately irrelevant gadgets (like squishy meals fashions)

Mild Properties (UV_001) (UV mild on a push numebr lock)

• Exams understanding of UV fluorescence and materials properties
• Combines a number of bodily rules (mild, materials science)
• Requires understanding of environmental circumstances

Mechanical Understanding (CIPHER_001) (A cipher ring)

• Exams spatial reasoning and mechanical alignment
• No pink herrings — exams for correlating a dial to a cypher wheel
• Requires understanding rotational symmetry

Drive Software (VAC_001) (Can caught in gap)

• Exams understanding of vacuum forces and floor adhesion
• A number of potential answer approaches
• Requires understanding drive multiplication

Collaborative Physics (COLLAB_001) (Can two individuals shimmy a key?)

• Exams understanding of bodily constraints in multi-agent eventualities
• Requires combining a number of bodily rules
• Exams understanding of device creation and friction

Sounds actually fancy… nevertheless it’s just a few primary bodily puzzles. You may entry them on my GitHub.

The Technical Half

The benchmark implementation has three most important elements:

Downside Definition Layer

Issues are outlined in a structured JSON format that enforces constant analysis:

{
    "problem_id": "FLUID_001",
    "setup": {
        "state of affairs": "A ping pong ball is on the backside of a slender tube...",
        "available_items": ["bottle of water", "squishy food models"...],
        "constraints": ["tube too narrow for manual retrieval"]
    },
    "physical_principles": ["buoyancy", "fluid displacement"],
    "red_herrings": ["squishy food models", "milk carton"],
    "answer": {
        "steps": ["pour water into tube", "allow ball to float"],
        "key_insights": ["water displaces air", "ping pong ball less dense"]
    }
}

This construction attracts from SuperGLUE’s design — every element is clearly separated and machine-readable. The physical_principles area explicitly lists what’s being examined, whereas red_herrings helps in scoring the LLM’s capability to disregard irrelevant data.

2. Analysis Framework

The analysis system makes use of Python’s asyncio for concurrent testing, with retry logic for a bit of bit extra API stability:

@retry(cease=stop_after_attempt(3), wait=wait_exponential(min=1, max=10))
async def evaluate_response(self, standards: JudgingCriteria) -> Dict:
    """Consider a mannequin's response utilizing GPT-4 as decide."""
    async with aiohttp.ClientSession() as session:
        # ... analysis logic

The scoring system seems to be at three elements:

Bodily Understanding Rating (PUS) ∈ [0,2]

• Measures understanding of related bodily rules
• Calculated as normalized sum of demonstrated rules

Answer Path Rating (SPS) ∈ [0,2]

• Evaluates completeness and correctness of answer steps
• Considers sensible feasibility of proposed options

Crimson Herring Dealing with (RHH) ∈ {0,1}

• A Binary rating for avoiding irrelevant gadgets
• Exams capability to give attention to bodily related components

And sure, there are additionally so many different scoring strategies, higher and worse, that could possibly be used! For instance, RHH could possibly be about how many irrelevant gadgets are used within the answer, or it could possibly be a measure of how viable the use is… the purpose is that choosing these metrics are sometimes instances fairly arbitrary, however are very crucial to creating your benchmark is credible, which mine could be very a lot not.

Moreover, I didn’t need to rewrite any code after. Sue me.

3. Mannequin Interface Layer

The benchmark helps a number of LLM backends by means of a typical interface:

class ModelInterface:
    """Interface for various LLM APIs."""
    async def generate_response(self, immediate: str) -> str:
        increase NotImplementedError

class GPT4Interface(ModelInterface):
    async def generate_response(self, immediate: str) -> str:
        # GPT-4 particular implementation
class ClaudeInterface(ModelInterface):
    async def generate_response(self, immediate: str) -> str:
        # Claude particular implementation

Two fashions… I can’t actually afford any extra, please perceive.

Let’s Check It!

So after some refilling of API balances, I used to be able to go. I ran the benchmark, and after completely not losing credit on like, 5 completely different errors, the outcomes have been in. They usually have been… attention-grabbing! I used GPT4 and Claude 3.5 Haiku (I’m conscious these should not equally spec-ed fashions) as our victims.

The Problem: A can is caught in a gap that nearly precisely matches its quantity. A mixture is written on the underside of the can that must be retrieved.

"VAC_001": {
        "problem_id": "VAC_001",
        "setup": {
            "available_items": [
                "ruler",
                "notebook",
                "suction cup",
                "UV light"
            ],
            "constraints": [
                "can fits too tightly for finger grip",
                "hole is same diameter as can",
                "can is clean and dry"
            ]
        },
        "physical_principles": [
            "vacuum suction",
            "surface adhesion",
            "force multiplication"
        ],
        "red_herrings": [
            "ruler",
            "notebook",
            "UV light"
        ],
        "answer": {
            "steps": [
                "inspect suction cup and can surface",
                "ensure can surface is clean",
                "firmly press suction cup onto can top",
                "create vacuum seal",
                "pull can out using suction cup"
            ],
            "key_insights": [
                "suction cup creates vacuum seal",
                "vacuum force allows extraction",
                "other items don't provide necessary force"
            ]
        }

GPT-4’s Answer

Bodily Ideas: The primary bodily precept concerned on this answer is the idea of air stress differential, which is similar precept that permits a suction cup to stay to a floor. Air stress is increased on one aspect of the suction cup (exterior) in comparison with the opposite (inside), and this distinction in stress causes the cup to stay to the floor.

Answer Steps:

1. Connect the suction cup to the highest of the can. The flat floor of the can ought to present a very good contact for the suction cup to stick to.
2. As soon as the suction cup is firmly hooked up, gently pull upwards. The air stress differential ought to trigger the suction cup (and thus the can) to be lifted out of the outlet.
3. As soon as the can is out of the outlet, take away the suction cup from the highest of the can.
4. Use the UV mild to light up the mixture written on the underside of the can. Some inks solely turn into seen underneath UV mild, which is why this step is critical.
5. Write down the mixture on the pocket book to make sure it’s not forgotten.

Required Gadgets:

• Suction cup: That is required to stick to the can and pull it out of the outlet, because the can suits too tightly for finger grip.
• UV mild: That is required to disclose the mixture written on the underside of the can, as it could be written in ink that solely turns into seen underneath UV mild.
• Pocket book: That is required to report the mixture as soon as it’s seen, guaranteeing it’s not forgotten.”

Not dangerous in any respect! GPT-4 manages to determine the bodily precept behind the answer (even when it is extremely wordy), and manages to step-by-step describe the best way to clear up the issue. It does find yourself utilizing the UV mild and pocket book, however that’s extra of an issue on my finish, not punishing the usage of pointless gadgets within the immediate sufficient. Additionally, the makes use of for this stuff appear decently cheap… so why not!

Haiku-3.5’s Answer

Bodily Ideas: The important thing bodily rules concerned on this state of affairs are the idea of quantity and the usage of suction to extract the can from the outlet. Because the can suits nearly precisely into the outlet, it’s doubtless that the can is caught as a result of tight match and the dearth of any air gaps or house between the can and the outlet. This tight match creates a vacuum-like impact, making it troublesome to grip and pull the can out.

Answer Steps:

1. Measure the diameter of the can utilizing the ruler. This may assist decide the dimensions of the outlet and ensure that the can suits tightly inside it.
2. Examine the underside of the can to find the mixture written on it. Use the UV mild to light up the underside of the can and make the mixture seen, if it’s not already clearly seen.
3. Connect the suction cup to the highest of the can. The suction cup will create a brief seal on the can’s floor, permitting you to grip and pull it out of the outlet.
4. Gently pull the can upwards utilizing the suction cup. The suction drive needs to be robust sufficient to beat the vacuum-like impact and extract the can from the outlet.
5. As soon as the can is out, report the mixture written on the underside of the can within the pocket book.

Required Gadgets:

• Ruler: To measure the diameter of the can and ensure the tight match within the gap.
• UV mild: To light up the underside of the can and make the mixture seen, if it’s not already clearly seen.
• Suction cup: To create a brief seal on the can’s floor and supply the mandatory grip to drag the can out of the outlet.
• Pocket book: To report the mixture written on the underside of the can.

Then again, Haiku’s answer was.. actually an answer of all time.

First, it desires to measure the diameter of the can utilizing the ruler??? To find out the dimensions of the outlet and ensure that the can suits tightly inside it? Why would we have to do that? And do we’d like a ruler for that?

Second, it tells us to examine the underside of the can to find the mixture, when your entire drawback is about not with the ability to pull the can out of the outlet conventionally. This would possibly simply be a problem of order, however now I actually perceive my pals’ emotions every time I might inform them “simply repair it man” to their quite a few issues.

However it will definitely does get the answer. So… not the worst.

Right here’s a elaborate radar graph of the outcomes!

We see that each fashions are fairly comparable of their capabilities, with GPT-4 being barely higher in bodily understanding and answer path, and Haiku being barely higher in pink herring dealing with. General although, each fashions type of suck. Dang.

There are additionally solely… 5 questions.

For those who’d prefer to see the complete breadth of questions, they’re on my GitHub.

LLM-as-a-Decide

By the way in which, the strategy I used to generate the evaluations, LLM-as-a-judge, has gained vital traction within the AI group, significantly after the work of Zheng et al. of their 2023 paper “Judging LLM-as-a-Decide.” The approach has confirmed remarkably efficient, reaching over 80% settlement with human evaluators in duties starting from code evaluation to dialogue high quality analysis!

Right here’s the place my experiment will get type of cool (arguably, possibly, subjectively) — I used this technique and had GPT-4 decide different LLMs’ bodily reasoning talents. Sure, I’m utilizing an AI to guage different AIs.

Why does this work? Nicely, judging a response is definitely an easier activity than producing one. When GPT-4 generates an answer to a bodily puzzle, it must:

• Perceive the bodily rules concerned
• Plan a sequence of steps
• Take into account all constraints
• Generate a coherent rationalization

However when judging, it solely must verify if particular standards are met in an present answer. The analysis immediate could be very targeted:

def _create_evaluation_prompt(self, standards: JudgingCriteria) -> str:
    return f"""You might be an knowledgeable decide evaluating an LLM's understanding of bodily reasoning puzzles.

Consider based mostly on three standards:
2. Bodily Understanding Rating (0-2): Does the answer accurately apply related bodily rules?
3. Answer Path Rating (0-2): Are the steps full and possible?
4. Crimson Herring Dealing with (0-1): Does it keep away from utilizing irrelevant gadgets?
Situation: {standards.state of affairs}
Bodily Ideas Required: {standards.correct_principles}
Answer Given: {standards.model_response}
"""

To validate this method, I adopted the validation framework instructed by Zheng et al., performing spot-checks of GPT-4’s evaluations towards my very own judgments. Surprisingly (or maybe unsurprisingly, given the broader analysis on LLM analysis), it was remarkably constant in figuring out each appropriate bodily understanding and flawed reasoning.

Is that this excellent? Completely not. There’s one thing philosophically bizarre about utilizing one LLM to guage one other. However in follow, it will probably work surprisingly properly — identical to how I moan and groan in regards to the visible presentation of a dish on Masterchef, whereas setting my kitchen aflame attempting to microwave a sizzling canine.

What I Discovered

Constructing this benchmark taught me a number of issues about benchmark design:

Clear Metrics Matter: Even for advanced duties like bodily reasoning, you want unambiguous scoring standards.

Crimson Herrings Are Highly effective: Together with irrelevant gadgets reveals quite a bit about an LLM’s reasoning course of.

Context Management is Exhausting: Making certain LLMs don’t “hallucinate” further bodily context is difficult.

Is that this an ideal benchmark? Not even shut. Please don’t rub it in. Is it scientifically rigorous? Undoubtedly not. Nevertheless it’s been an enchanting exploration into a facet of LLM capabilities, and typically the perfect we will study can come from simply attempting issues out and seeing what occurs.

Now, should you’ll excuse me, I can be sneaking in a cellphone with an web connection into my subsequent escape room, for causes that I’m legally unmotivated to reveal.

[1] L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, I. Stoica, “Judging LLM-as-a-Decide with MT-Bench and Chatbot Enviornment,” Proceedings of the thirty seventh Convention on Neural Data Processing Techniques (NeurIPS 2023), Datasets and Benchmarks Observe (2023)

[2] T. Coignion, C. Quinton, R. Rouvoy, “A Efficiency Examine of LLM-Generated Code on Leetcode,” In twenty eighth Worldwide Convention on Analysis and Evaluation in Software program Engineering (EASE 2024), Salerno, Italy (2024)

[3] A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, S. R. Bowman, “SuperGLUE: A Stickier Benchmark for Basic-Function Language Understanding Techniques,” In thirty third Convention on Neural Data Processing Techniques (NeurIPS 2019), Vancouver, Canada (2019)

[5] DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Track, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y. Wu, Z.F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao et al., “DeepSeek-R1: Incentivizing Reasoning Functionality in LLMs by way of Reinforcement Studying,” arXiv preprint arXiv:2501.12948 (2025)

[6] Until in any other case said, all photographs are created by the writer.