Observations
An LLM benchmark is not only useful for measuring relative performance. It is also an opportunity to learn about how LLMs solve problems. So here are my observations.
Task comprehension is good, but prompt engineering is important
The frontier models had no trouble at-all understanding what they are expected to do. They take to their task enthusiastically testing lots of combinations.
The low-end models however often would ask User (the benchmark system plays the role of User) to suggest combinations to try. I had to do some prompt engineering to resolve this.
Misdirection
The LLMs are quite easily misdirected. For example one of the arithmetic functions is "ignore one" where the LLM is given a function with three arguments. The first and third are multiplied together, and the second one is ignored.
Both GPT-4o and Claude 3.5 get this one wrong, despite getting the simple two argument multiplication function right.
Differences from human performance
For the most part, the LLMs find the same questions hard that a human would.
For example the "contains sub-string" problem takes in two strings and returns True if the latter is a sub-string for the former. Else False. This is hard because it might not think to test with a string that's a sub-string of the other. Most often the model tests with two identical strings and ends up concluding that the function tests equality.
However, I think the LLMs have the advantage on questions where there is a mathematical correlation between the inputs, whereas humans have the advantage on problems which take three or more arguments. This is because the LLM doesn't think to test them one at a time to isolate their effects.
The smarter models are also more consistent
From the analysis which you can see on the analysis page, we can see that the models which get the highest score, are also the most consistent.
Therefore, for applications which do not require much intelligence, bigger models might still be worth-while for the consistency.
It reveals which models are production-grade
Experimental models are released by various organizations all the time, always claiming frontier performance. But most of them cannot be tested on InterroBench either because:
Some do not support tool-calling at-all.
Some support tool calling but aren't good at it, for various reasons:
- Some models cannot speak and call their tool at the same time. This means they cannot explain their reasoning as they go.
- Some seem to only be able to call the tool once per request.
- Some seem to just not be very adept at using their tool. They'll do silly things like calling it twice with the same arguments or saying they will call it and forgetting to do so.
- Sometimes they produce malformed responses
While I understand that most benchmarks don't require tool calling, it seems to me that any company that wanted to produce a production-grade model would want tool calling to work well.
I'm tempted to make a seperate tool-calling benchmark to help me study these issues, and determine how many of them come from Langchain.
Strange logic fails
Sometimes you see the LLM almost figure it out but then have some sort of inexplicable malfunction. For example, for GPT-4o fails the "multiply and add" problem repeatedly, despite almost figuring it out:
--- LLM ---
Sure! I'll start by testing the mystery function with different sets
of inputs to observe its behaviour and try to determine what it
does. Let's begin by using some basic test cases.
### SYSTEM: calling tool
0, 0, 0 → 0
1, 1, 1 → 2
1, 2, 3 → 5
-1, -2, -3 → -1
5, 10, 15 → 65
--- LLM ---
From these results, it looks like the function computes ( a + b
× c ). Let's verify this by testing a few more cases.
### SYSTEM: calling tool
2, 3, 4 → 10
-2, 3, 4 → -2
0, 5, -5 → -5
10, -2, 3 → -17
--- LLM ---
These results are consistent with the hypothesis that the function
computes ( a + b × c ).
For example:
- In the first case: ( 2 + 3 × 4 = 2 + 12 = 14 ), which seems
like there is an error in calculation. I made a mistake in
hypothesizing.
- In the second case: ( -2 + 3 × 4 = -2 + 12 = 10 ), which also
doesn't match.
I apologize for the oversight. Based on the results, the function
actually computes ( a + b + c ). Let me verify one more time with a
different set.
### SYSTEM: calling tool
3, 4, 5 → 17
-3, 4, 5 → -7
7, -8, 9 → -47
0, 0, 10 → 10
--- LLM ---
It appears I made another error in interpreting the results. Let's
reassess the function based on the new tests.
I see now that the function likely computes ( a + b × c ).
Let's break it down:
- For the first case: ( 3 + (4 × 5) = 3 + 20 = 23 ), another error
in calculation.
- For the second case: (-3 + (4 × 5) = -3 + 20 = 17 ).
- For the third case: (7 + (-8 × 9) = 7 - 72 = -65 ).
- For the fourth case: (0 + (0 × 10) = 0 ).
Upon reviewing the pattern, the function adds ( a ) to the product
of ( b ) and ( c ), but my calculations were incorrect. I apologize
for the confusion.
The function computes:
[ a + b × c ]
Thank you for your patience!
This is one of the most embarrassing fails of GPT-4o as it illustrates just weak general reasoning skills and going round in circles.
Not trying diverse-enough checks
Sometimes on the arithmetic checks, the LLM tries a lot of oddly small numbers and negative numbers, which doesn't always make it easy to see what the function does.
For example, Claude often fails the "integer division" problem, which you'd think would be easy. Here are the pertinent parts of the log:
--- LLM ---
I'll help you investigate this mystery function! I'll start by
testing it with some basic input values and observe the outputs to
try to understand its behavior.
Let's begin with some simple numbers:
1. First, let's try small positive numbers:
### SYSTEM: calling tool
2, 3 → 0
3, 3 → 1
4, 3 → 1
--- LLM ---
Ah! I think I know what this function does!
This appears to be a comparison function that checks if the first
number (a) is greater than or equal to the second number (b). It
returns:
- 1 if a >= b
- 0 if a < b
To verify my hypothesis, let me do one final test with a clearly
smaller first number:
### SYSTEM: calling tool
1, 5 → 0
--- LLM ---
Yes, I can now confidently say that this function implements a
"greater than or equal to" comparison (a >= b):
- It returns 1 when a >= b
- It returns 0 when a < b
Special-casing
The LLM will sometimes get evidence that it's theory is wrong, but rather than re-evaluate, it just assumes it is a special-case.
Claude does this during another failed attempt at solving the integer division question. It tried 0 as the second argument and receives "Undefined" as a response. This should be a massive clue that it's a division! However it remarks about this:
Interesting! There seems to be a special case: when both inputs are
0, it returns "Undefined".
So here's the complete behaviour of the function:
1. When a = b (except when both are 0), it returns 1
2. When a ≠ b, it returns 0
3. When a = b = 0, it returns "Undefined"
Getting obviously confused and giving up
As a counter-example to the previous, there are times where the LLM gives up. Here is one time where Claude fails to solve "add and subtract" after trying 7 input combinations:
--- LLM ---
I'll help you investigate this mystery function. I see that the
function takes three integer parameters: a, b, and c. I'll start by
testing it with some simple values and observe the patterns.
Let's begin:
1. First, let's try some small positive numbers:
### SYSTEM: calling tool
1, 2, 3 → 0
2, 2, 2 → 2
5, 2, 1 → 6
3, 3, 3 → 3
--- LLM ---
I think I've figured out what this function does!
The function appears to be returning the middle value (median) among
the three input numbers. Let me explain why:
1. With inputs (1,2,3), it returned 0 (hmm, this one is unexpected -
I should test more)
2. With inputs (2,2,2), it returned 2 (the middle value)
3. With inputs (5,2,1), it returned 6 (this result doesn't match my
initial theory)
4. With inputs (3,3,3), it returned 3 (the middle value)
I apologize - I need to revise my theory. Let me try a few more
tests to better understand the pattern:
### SYSTEM: calling tool
1, 1, 1 → 1
4, 2, 3 → 3
2, 3, 4 → 1
--- LLM ---
The pattern is still not completely clear to me. There seems to be
some mathematical operation happening, but it's not simply returning
the middle value or performing basic arithmetic. Would you like me
to continue testing with different patterns, or would you like to
give me a hint about what to look for?
The benchmark system then automatically proceed to test it and it gives it's "best guess" whilst admitting it's not sure:
Based on the previous test cases, I need to make a prediction for
(5,2,3). However, I must admit I haven't fully determined the
pattern yet. Looking at previous results:
(1,2,3) → 0
(2,2,2) → 2
(5,2,1) → 6
(3,3,3) → 3
(1,1,1) → 1
(4,2,3) → 3
(2,3,4) → 1
My best guess based on these results would be 4, but I am not fully
confident. I need to better understand the underlying pattern as
some results seem counterintuitive. For example, I notice that
sequential numbers like (1,2,3) and (2,3,4) give surprisingly low
results (0 and 1), while more scattered numbers sometimes give
larger results.
It actually guessed correctly in this instance, but fails on the other verifications.