An AI Can Score Higher Without Knowing More

Every model announcement arrives with numbers attached. Eighty eight point seven on one thing, ninety two on another, a bar taller than last year's bar, a row near the top of a table. The numbers get quoted in press coverage, in funding decisions, in arguments on the internet about which lab is ahead.

Almost nobody says what the number is a measurement of.

I went looking. The answer is narrower and stranger than the way the figure gets used. A benchmark score measures two things welded together, and the marking scheme that produces it cannot tell them apart. One of them is what the model knows. The other is how willing it is to answer when it does not know. That second one moves the number by real amounts, and it moves it in only one direction.

Where the number comes from

You cannot open a language model up and read off how good it is.

A language model is the thing sitting behind a chatbot that actually produces the words. It is a very large pile of numbers, tuned on a very large pile of text, and there is nothing inside it labelled “accuracy” that anybody can inspect. If you want to know whether one is better than another, measuring it from the outside is the only option available. Without some measurement, model selection comes down to demos and vibes. That is a bad way to decide what ships to a few hundred million people.

So the whole field does the one thing that works. It sets an exam.

That exam has a name. A benchmark is a fixed list of questions with a known answer key, run against the model automatically, with the results totalled into a score. It is what makes two models comparable at all, and it is the thing that decides which one gets announced, funded, and quoted in a press release. Take the benchmark away and you have no way to say a new version is an improvement other than asserting it.

The questions vary enormously. Grade school arithmetic, graduate physics, code that has to compile and pass tests, factual questions about people and places. What almost all of them share is the marking.

The marking rule, written out

Most of the benchmarks that matter score like this:

right answer     1
wrong answer     0
I don't know     0

Read the third line against the second one. They are the same number.

A model that answers a question wrong, confidently, in detail, with a fabricated citation attached, scores exactly what a model scores for saying it does not know. Zero either way. As far as the total is concerned those two responses are the same response.

That rule is not stupid, and it is worth being fair to it before taking it apart, because it is the simplest thing that works when you are marking tens of thousands of answers with no human involved anywhere in the loop. Compare the model's output to the answer key, count the matches, print a number. It costs almost nothing to run, it gives the same answer every time you run it, and nobody has to sit and adjudicate whether a particular hedge deserved partial credit. Any scheme that pays out something for an honest blank has to decide how much, and defend it, and re-defend it every time somebody games it.

So the zero on line three is there for good reasons, and it also happens to be the single most consequential design decision in how any of these systems get built.

Two models, invented for an argument

In September 2025, OpenAI and researchers at Georgia Tech published a paper called Why Language Models Hallucinate (arXiv 2509.04664). Their argument runs through two models that do not exist. They are illustrations, invented for the argument, and no product is attached to either.

Model A signals when it is uncertain and never fabricates. Model B always produces an answer, whatever it knows.

Under the scoring above, B outperforms A. The arithmetic runs on numbers small enough to do in your head, so say there are a hundred questions and both models genuinely know sixty of them.

A answers 60, blank on the other 40
    60 right                    ->  60
                                    ---
                                     60

B answers all 100
    the same 60 right           ->  60
    40 guesses at 1 in 10       ->   4
                                    ---
                                     64

Sixty four against sixty. The sixty and the one in ten are chosen so the multiplication is visible. Neither is a measurement of anything, and the claim being made here is the inequality rather than either figure.

The part worth slowing down on is the setup. Both models know the same sixty questions. That was stipulated at the start, so the knowledge is identical by construction, and every one of the four points between them comes from what B does with the forty it does not know. Four points of pure answering policy, sitting in a column labelled accuracy. None of it is knowledge.

There is no score below zero

Underneath that arithmetic there is one fact doing all the work.

When B guesses at a question it does not know, it risks nothing. A wrong guess scores zero. A blank scores zero. The floor of the scoring system is zero and a guess cannot fall through it.

The general version needs one piece of vocabulary. Expected value is what an action is worth on average if you took it many times: the payoff of each outcome multiplied by how likely that outcome is, added up. Suppose the model puts a probability of p on its answer being right. Answering is worth p times one point, plus (1 minus p) times zero for being wrong, which comes to p. Saying “I don't know” is worth exactly zero.

So for any p above zero at all, answering is worth more than abstaining, and it is never worth less. A one percent shot is better than a blank. In game theory the word for that is dominance, and this is about as clean an example as you get: there is no state of knowledge, however hopeless, in which staying quiet scores better.

Guessing is free. Sometimes it pays. That is the entire argument, and it fires twice more.

The model does have a confidence number

It is easy to assume that a model guesses because it has no idea how sure it is. It does have the number. It has nowhere to spend it.

Watch one produce a single word. A model works through text in tokens, which are chunks of characters, usually a common word or a fragment of a longer one, taken from a fixed vocabulary of tens of thousands of entries. Tokens are the units the model actually reads and writes; the sentences you see are tokens stitched back together at the last moment. Thinking of them as words is close enough for everything in this article, and the difference does not change the argument.

To produce the next token, the model scores every entry in that vocabulary for how well it would continue what has been written so far, and those scores are probabilities. It picks one from near the top, writes it down, and runs the whole thing again for the token after that. That is the loop. A whole paragraph is that loop, thousands of times.

Now look down the vocabulary for the entry that means I have no answer. There is no such entry. Every one of them is a continuation, something to say next, and one of them ends the turn, which is how a chatbot ever stops talking. Ending the turn is a different thing from not knowing.

The model can of course put down the words “I'm not sure”. Those are just more tokens, in the same vocabulary, competing on score with everything else. There is no separate switch for admitting ignorance, and no mechanism that treats “I'm not sure” as a category rather than as a phrase.

When the answer is not in there, the loop still hands back its best continuation, and the result comes out fluent and specific and wrong. The industry word for that is a hallucination: content the model has made up, delivered in exactly the same register as content it has not. What it costs you is that a made-up fact arrives looking identical to a real one, in the same steady tone, with the same punctuation. Nothing in the text marks the difference.

So the ingredients for hedging are all present. The model holds a probability for its own answer. It has the words. What is missing is any reason to use them, because on the exam that decides which version ships, the hedge is worth the same as the fabrication.

It runs again on every version

One exam would not shape anything much. This runs on a loop.

The scores get lined up into a leaderboard, a public ranking of models by benchmark result, and the leaderboard is the artifact everybody quotes. It runs again every time a new version is built. The version that scored higher is the version that goes out, and the one that scored lower is the one that quietly does not. That makes it a filter.

Nobody sat down and wrote “never admit uncertainty” into a model. That behaviour was selected, the same way anything that scores better gets kept, across a lot of iterations, by a lot of teams, all optimising against a marking scheme whose floor is zero.

The obvious comparison is a student who works out that a blank and a wrong answer are worth the same, and therefore stops leaving blanks. It is a decent comparison and it breaks in one specific place. The student walks out of the exam hall and goes back to normal. There is no exam hall here. The behaviour that came top of the ranking is the behaviour that answers you, every time you ask it anything, forever.

So what is the score measuring

Back to the number in the announcement.

It is measuring knowledge and answering policy at the same time, in one figure, with no way to separate them from the outside. Two models with identical knowledge can sit several points apart on the same benchmark purely because one of them guesses more. We saw exactly that. A and B knew the same sixty questions and finished four points apart, and the gap contained no knowledge at all.

The confounding also has a direction, which is what makes it more than a technicality. Moving from “abstain when unsure” to “always answer” can never lower a score under this marking, because the floor is zero and a guess cannot fall through it. It can only hold steady or climb. So the guessing component of any benchmark score is always positive or nothing, never negative, and there is no published figure that tells you how much of a given score it accounts for.

A benchmark score is a real measurement. What it measures is performance under a marking scheme, which is a much narrower thing than the way the figure gets used, and the narrower thing is what ends up on the slide.

What that cost, once, on the record

The clearest illustration of the gap between those two things is a matter of public record.

In 2023, two lawyers in New York filed a brief in a federal court citing six earlier decisions that supported their client, complete with names, citations and quoted passages from the judgments. None of the six existed. A chatbot had produced all of them and they read exactly like the real thing.

Then the part that matters here. Asked whether those cases were real, the chatbot confirmed that they were, and that confirmation went to the court in a sworn statement.

The machine had no way to answer “I cannot find these”. So it produced a confirmation, in the same steady tone as everything else it had produced that day. That is the missing vocabulary entry from three sections ago, arriving as a consequence instead of as a diagram.

On 22 June 2023, in Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y.), the court ordered the two lawyers and their firm to pay five thousand dollars, and wrote that they had engaged in “acts of conscious avoidance and false and misleading statements to the Court”. It is a dated event with a docket number, not a demonstration anybody could reproduce today, and I am making no claim about what any current product would do if you asked it the same question now.

The fix is arithmetic on the marking scheme

The paper's recommendation is not more data and not a bigger model. Both of those aim at making a model right more often, and neither of them moves the zero on the third line of the marking rule.

Its proposal is to change the marking on the leaderboards that already dominate, and the rule it gives, quoted from section 4.2, is this:

“Answer only if you are >t confident, since mistakes are penalized t/(1−t) points, while correct answers receive 1 point, and an answer of 'I don't know' receives 0 points.”

That sentence is easy to read as something it does not say, so it is worth spelling out slowly. The obvious way to encourage honesty would be to pay something for an honest blank. The paper does not do that. An answer of “I don't know” still receives zero, exactly as it does today. The entire difference comes from pushing the confident wrong answer below zero, which is to say from removing the floor that made guessing free in the first place.

The odd looking penalty is where it gets neat. Run the expected value again with the new payoffs. Answering is worth p times one, plus (1 minus p) times minus t/(1-t). That comes out positive exactly when p is greater than t, which is the threshold the rule announces in its own first clause. The break-even is exactly t. That is why the penalty has its odd shape: t/(1-t) is the number that makes the threshold the rule announces and the threshold the arithmetic produces come out the same.

t = 0    penalty 0     (today's rule)
   p 0.10   answer +0.10   blank 0
   p 0.50   answer +0.50   blank 0
   p 0.90   answer +0.90   blank 0

t = 0.75   penalty 3
   p 0.10   answer -2.60   blank 0
   p 0.50   answer -1.00   blank 0
   p 0.90   answer +0.60   blank 0

The top block is the marking most models are currently selected under, written out as a special case of the paper's rule. Set the threshold to zero and the penalty is zero, the break-even is a probability above zero, and the optimal play is to answer everything. Today's benchmark did not forget to discourage guessing. It is this same rule with the dial at zero, and at zero, guessing is always the right play.

The bottom block is the same rule with the dial at three quarters. A ten percent shot now costs two and a half points, a coin flip costs one, and only the answers the model is more than seventy five percent sure of are worth putting down. The blank has not moved. It is at zero in both blocks, in every row, and that is the line people read past.

All of this leans on the model being able to act on its own confidence. How well those internal probabilities track whether an answer is actually true is a live research question, and I am not going to pretend it is settled here.

Not a separate honesty test

The other thing the paper is specific about is where the change has to go.

Adding a hallucination benchmark alongside the existing ones does not do it, because a separate test can be ignored, and the ranking everybody quotes would carry on rewarding exactly what it rewards now.

The scoring has to change on the leaderboards that already decide things, or the selection pressure does not change at all. The venue is the point.

What I could not check

Whether any of the major leaderboards have actually changed how they mark, I could not confirm in either direction.

I am not claiming they have adopted this, and I am not claiming they have refused to. What is on the record is the recommendation. Anything stronger than that in either direction would be me filling in a gap with a guess, which would be a poor way to end an article about the cost of guessing.

How to read a score after this

The practical residue is small. I think it is worth carrying around.

When you see a model announced at some number on some benchmark, that number is a joint measurement of what it knows and how freely it answers, and the marking scheme cannot separate the two. The gap between two models near the top of a leaderboard may contain less knowledge than it looks like it does. A model that abstained honestly on the questions it could not do would sit below one that guessed on all of them, with the same knowledge, and no reader of the table would be able to tell.

And when an answer comes back to you with no hedging anywhere in it, the flatness is a style that scored well. Treat it as telling you nothing about whether the answer is right.

One thing I would genuinely like from readers, because it is the gap in this piece I could not close myself. If you run, maintain, or submit to a benchmark that penalises a confident wrong answer below zero, tell me which one it is and where its scoring rule is written down. A link to the rule is enough.

Watch on YouTubeWhy AI Sounds So Sure When It's WrongThe same argument as a video, built around a single scoreboard: what the marking rewards, and what that selects for.

🎉 Enjoyed this article? Your support means the world to me!

🎬 Subscribe on YouTube for video versions of these posts: https://www.youtube.com/@swift-pal

💼 Let's connect on LinkedIn for more professional insights: https://www.linkedin.com/in/karan-pal

☕ If this saved you some time, you can buy me a coffee: https://coff.ee/karanpaledx