Do the different models find the same problems challenging?
The first analysis I wanted to do was, do the LLMs "agree" on which problems are difficult? Or are different problems difficult to different LLMs?
To investigate this I:
- ordered the problems by difficulty, based on how many LLMs got that question right
- plotted the problems each LLM got right and wrong as a bar code
Here are the results, starting from the weakest LLMs:
grok-beta
llama3-groq-70b-8192-tool-use-preview
command-r-plus
gemini-1.5-pro
gpt-4o-mini-2024-07-18
gpt-4o-2024-08-06
claude-3-5-haiku-20241022
claude-3-5-sonnet-20241022
What does this mean?
It does superficially look like there may be some problems which are idiosyncratically difficult for certain LLMs. However I think it's mostly just that each LLM has a sweet-spot of difficulty where they have an even chance of getting them right or wrong.
We can analyse this since each LLM had 4 attempts at each problem. The following plots show:
- problems it always gets right in black
- problems it usually gets right in purple
- problems it gets right half the time in red
- problems it usually gets wrong in orange
- problems it always gets wrong in white
grok-beta
llama3-groq-70b-8192-tool-use-preview
command-r-plus
gemini-1.5-pro
gpt-4o-mini-2024-07-18
gpt-4o-2024-08-06
claude-3-5-haiku-20241022
claude-3-5-sonnet-20241022
In previous versions of this benchmark, I used a "best of 5" for each problem to quantize the result for each problem to a pass or fail. However from version 6 I just run each problem multiple times and give the model a point for every correct attempt. It works out the same on average.
The funny thing is about all this, if you are examining a human, you can't actually test each problem multiple times. Because there is no way to wipe a human's memory between each attempt.
Makes one wonder exactly how deterministic our brains actually are. If someone does an exam, and they get a certain question right, did they have a 100% probability of getting it right? Maybe they only had a 50% probability of getting it right. Or even 10%.
It's hard to measure since you'd need to be able to copy a person several times and get them all to take the same exam at the same time. Cloning wouldn't be good-enough because clones could have different life experiences or had something different for breakfast. It would need to be a true copy like maybe a transporter accident in star-trek.
How thorough are the models in testing their function?
Do models that spend longer investigating their function get a higher score?
We keep metrics on how many times each LLM calls it's tool so we can easily check that.
Generally yes, we are seeing higher scores with more tool calls.
Poor little grok-beta often only calls it's tool twice. It has a lot of errors trying to call it's tool so often it's testing is often cut off short which is sad.
The right-hand side of the chart is dominated by OpenAI and Anthropic. OpenAI's GPT-4o and GPT-4o-mini call their tool around 1500 times whereas Claude Haiku and Sonnet only around 1000.