Why Is AI So Expensive to Run?

Type a question into a search box and the answer is already there. Links, instantly, before your hand has left the keyboard. Type the same question to an AI and something different happens: the words come out one at a time, at a readable pace, like there is somebody on the other end typing them.

Both of those answered your question, but only one of them cost real money to produce, and the gap between what the two of them cost is not a small one.

The reason has nothing to do with one machine being newer or bigger or cleverer than the other. They did completely different work, and the difference in the work is the whole story.

Here is the answer before the explanation, because I would rather you had it now than in eight minutes.

A search looks something up. An AI makes something. Looking things up is cheap because the expensive part already happened, once, years ago, and every person on Earth who searches is splitting that bill. Making things is expensive because it happens again, from nothing, for every single word, and nobody else on Earth shares any part of it with you.

Underneath that sits one question, and it is the only thing you need to carry through the rest of this article. How many times does this work have to happen, and how many people is it shared with? Everything below is that question asked four times, and the answers to it are what set the price.

What a search is actually doing

Start with the cheap machine, because most people have never been told what it does.

At some point a program went and fetched the web. Not a corner of it, the whole thing, hundreds of billions of separate pages. And as it fetched, it wrote something down. For every word it saw anywhere, it kept a list of which pages contain that word.

That thing it wrote down is called an index, and it is worth slowing down on because it is doing all the work here. It is the same idea as the index in the back of a textbook. The book is long, the index at the back is short, and the index exists so that you never have to open the book at page one and start reading to find what you want. You look up the word, you get the page numbers, you go straight there. Google's version is the same object, built for every word in every language, pointing at pages instead of at page numbers.

Each row has a name of its own. It is called a posting list, and it is exactly what it sounds like: one word, plus every place in the world that word turns up. The word jet has a posting list. So does engine. So does every word you have ever typed into a search box.

Now you type your question. Three or four words.

The machine does not go and look at the web. The web is not involved at all, and this is the part that surprises people. It pulls out the posting lists for your handful of words, finds the pages that appear in all of them, and hands those back. Three rows, out of every word in every language it has ever seen. Everything else in that enormous structure stays untouched.

It does not even read all of those three. Posting lists are built with shortcuts inside them, so the machine can jump forward over long stretches it already knows it does not need.

Then the part that actually makes it cheap. Google says it handles more than five trillion searches a year, which works out at around fourteen billion a day, and every single one of those is reading the same index that was built once. The crawling happened once, the organising happened once, and fourteen billion questions a day are splitting the cost of it, which is why the bill for your particular search is a rounding error on a rounding error.

The expensive part of an AI also happened once

Now the other machine. And here is where almost everybody's intuition goes wrong, including mine when I started reading about this.

An AI model is a file. That is genuinely all it is, a file sitting on a disk somewhere, and inside it one very long list of plain ordinary numbers. Those numbers have a name. They are called weights, and there are billions of them. Each one is a setting that got nudged very slightly into position during the process of building the model, and that process is called training.

There is no text in that file. Nothing in there is a sentence or a page or a fact you could go and look up, and none of it is an answer to anything. That part takes a while to sit properly in your head. It is numbers.

Training is the famously expensive thing. It is months of computation on enormous amounts of hardware, and the numbers attached to it get quoted in the hundreds of millions of dollars. So the obvious guess is that this is what you are paying for every time you send a question.

It is not, and the reason why is the same reason a search is cheap.

Training happens once. Then that file gets copied, and it answers questions for millions of people, for months or years, until it is replaced. It is the crawl, and it is the index being built: one enormous cost divided by a colossal number of uses, which is exactly the arrangement that makes a Google search cost nothing.

So if training were the difference, an AI answer would be cheap too. The difference is somewhere else, and it is in what happens after the file exists.

And then the expensive part happens again

To produce one word, the machine takes everything written so far, your question and whatever it has already said, and multiplies all of it through those numbers. Through all of them, from the front of the file to the back of it.

What falls out the other end is one word. Not the answer. One word of it.

Then it does the entire thing again for the word after that. And again. Once per word, all the way to the end of the answer.

Those words have a name as well, and you will meet it on every pricing page you ever look at. They are called tokens, and a token is the chunk of text these systems actually count in, which is usually a common word, a piece of a longer word, or a mark of punctuation. English runs a bit under one word per token, so when I say words and a screen says tokens, we mean nearly the same thing. When you hear that something costs a dollar per million tokens, that is what is being counted.

And none of this is shared with anybody. The index is built once and reused fourteen billion times a day. Your answer is assembled for you alone, word by word, and then it is thrown away. The next person asking the identical question gets the whole thing manufactured again from the beginning.

So there is the one question, asked four times. Crawling happened once and everybody shares it, training happened once and everybody shares it, reading the index costs a thin slice of it per question, and reading the model costs all of it, per word, for one person.

The part I did not expect

Put the two objects next to each other and it goes strange.

Google says its index is well over a hundred million gigabytes. A very large AI model is a few hundred gigabytes. One published model has 671 billion numbers in it, which lands somewhere around seven hundred gigabytes of file.

So the index is more than a hundred thousand times bigger than the model.

And the small one is the expensive one.

The big object gets read a tiny fraction at a time, once per question. The small object gets read completely, and then completely again, and again, once for every single word coming out of it.

Size was never the variable here. What matters is how much of the thing you have to go through to answer one question, and how many separate times you are forced to go through it.

I went looking for a number to put on the search side of that, some published figure for how many bytes a single Google query actually reads. There isn't one. Nobody publishes it. So I am not going to invent a figure, and the honest version of that sentence stays qualitative: a search reads a thin slice of an enormous object, and skips most of the slice.

A hundred terabytes, and a quarter of a watt-hour

The AI side we can actually do the arithmetic on.

Say a three hundred word answer, which is a reasonable reply to a reasonable question. Three hundred words means going through the model three hundred times, so take a model of around three hundred and fifty gigabytes and you have moved somewhere near a hundred terabytes of numbers to produce one reply.

A hundred terabytes is hard to feel, so put it against the object we already have, which is a search index of a hundred million gigabytes. A hundred terabytes is a hundred thousand gigabytes. Which means one AI answer, for one person, pushes roughly a thousandth of Google's entire index through a chip.

One search reads three rows and skips most of them.

That arithmetic is for a straightforward model that reads all of itself for every word. The largest models now have a trick for reading less of themselves, and it comes with numbers at the end of this article.

Now the other direction, because this is the place the loud version of this story gets it wrong.

That is not a frightening amount of electricity, because Google measured the median text prompt to its own model at 0.24 watt-hours, and the comparison they chose for it is watching television for under nine seconds. It also came with 0.03 grams of carbon dioxide and about a quarter of a millilitre of water.

One question is nothing. It was never about one question, and I will come back to that at the end with the multiplication.

Why the price list is shaped the way it is

Once you have seen the split, you cannot look at an AI pricing page the same way again.

Reading your question is one pass through the model, no matter how long the question is. A short question, a long question, an entire document pasted in. The machine can work through all of your input at the same time, so it costs one trip through the file.

Writing the answer is one pass per word.

So the two halves get billed at very different rates, and the gap is not subtle. One model, on one day, on a public price list: twenty-five cents per million tokens going in, ninety-five cents per million coming out. Prices move constantly, so the number worth holding on to is the ratio rather than the cents, and on that day output cost close to four times what input cost.

Both sides are still counted per token, so a longer question does cost you more than a short one. What it does not cost you is a fresh trip through the model, because all of your words ride through it together, and that is the whole reason each one of them is priced so far below a word of the answer.

You are not really paying for your question. You are paying for the answer.

Which explains the thing everybody noticed about the reasoning models, the ones that work through a problem before they answer. Those working-out steps are words too. The machine writes them one at a time, one full pass through the model each, and most of them are never shown to you. OpenAI's documentation says it plainly: hidden reasoning tokens are billed as output tokens.

You pay for words you never read.

Two things I simplified

I have been describing this as though you were the only person in the world talking to the machine. That is where I owe you a correction, and there are two of them. Both are the industry attacking exactly the cost this article is about, which is the strongest evidence I can offer that the mechanism is the real one.

The first is batching, and it starts from the fact that a server is not answering you alone. It is answering hundreds of people in the same moment, and it pulls the model through once for all of them together. Every one of those conversations needs the identical numbers, so one trip through the file feeds hundreds of answers instead of one. The cost of that read gets split across everybody in the batch, and the real per-person figure is far below the naive one.

Look at what batching actually is. It is sharing, put back: the same arrangement that makes the search index cheap, bolted on to the one place an AI answer could not get it by itself.

The second correction is bigger. I told you the machine reads the whole model for every word, and for a straightforward model that is exactly what happens. The very large ones now split themselves into specialists, groups of numbers that handle different kinds of material, and they wake only a few of those groups for any given word.

That published 671 billion number model uses about 37 billion of them for any one word. Eight specialists out of 256.

Which sounds like it demolishes everything above, and it does not, for two reasons. Thirty-seven billion numbers per word is still thirty-seven billion numbers per word, read from memory, for one word, for one person. And all 671 billion of them have to be sitting in memory ready to go, because the machine has no idea which specialists the next word will need until it gets there.

Then notice why either trick exists. Both were invented to attack the cost of dragging a model through a chip for every word it writes. Nobody redesigns a machine this thoroughly to dodge a problem that is not real.

It works, too, because Google says the energy behind one of its own prompts fell thirty-three times in a single year.

Three sentences, and then the multiplication

An index is built once and read a fraction at a time, by everybody, forever. A model is read again for every word it writes, for one person, and then that work is thrown away. So your bill tracks how much it says back to you, not how much it knows.

That is why a quarter of a watt-hour is worth anybody's attention.

Fourteen billion searches happen every day. If an AI answer sat on top of every one of them, which is roughly the direction all of this is heading, a quarter of a watt-hour becomes more than three million kilowatt-hours a day. The number per question is tiny. The number of questions is not.

One thing I would like from you, and it takes about a minute. Open the usage page of whichever AI you pay for and find your input and output token counts for the month. Then tell me the ratio between them.

I want it because the whole argument above says your bill is set by the output side, and I cannot check that against how real people actually use these things. Somebody pasting long documents in and getting short summaries out has a completely different bill from somebody asking one line and getting an essay. The pricing tells you what the two halves cost. It does not tell you which half people are actually buying, and about forty of you could answer that in a comment.

🎉 Enjoyed this article? Your support means the world to me!

🎬 Subscribe on YouTube for video versions of these posts: https://www.youtube.com/@swift-pal

💼 Let's connect on LinkedIn for more professional insights: https://www.linkedin.com/in/karan-pal

☕ If this saved you some time, you can buy me a coffee: https://coff.ee/karanpaledx