Why Does AI Write One Word at a Time Even on a Fast GPU?

You've watched this happen. You ask a question, and the answer comes back one word at a time. Not all at once. At a pace, steady and even, like someone typing on the other side.

That pace is not the machine thinking.

It's a physical limit, and it isn't the one most people will point you at. Something in the hardware can only go so fast, so the words come out at exactly that speed. Which raises a fair question, because this is supposed to be running on the fastest consumer hardware we know how to build.

It runs on a GPU, a graphics processor. It's the chip that exists because somebody wanted to draw video games, and it now does almost all of the artificial intelligence on Earth. So why that chip, and if it's so good at this, why is your answer still arriving one word at a time?

Here's the answer up front rather than at the end. A GPU is good at AI because AI is one tiny piece of arithmetic repeated an absurd number of times, and a GPU is a machine built to do one tiny piece of arithmetic an absurd number of times. That part is true and it's why GPUs got the job. But the speed you actually see has almost nothing to do with how fast those cores can compute. It's set by how fast the chip can carry the model's numbers to them, and that is a different number entirely.

The rest of this is why that's true, and the four things it explains that you've probably already noticed without having a reason for.

The answer everyone gives, and it's correct

Your CPU is the main processor, the one that runs everything else on the machine. Inside it are cores, which are the parts that actually do the work. A CPU has a handful of them, and they're big. Each core is clever, each can handle complicated work with a lot of decisions in it, and each one works through its jobs in order.

A GPU made the opposite trade. Thousands of small cores, not one of them clever, all of them capable of doing the same simple thing at the same moment.

That second design is useless for most software. It's close to perfect for this one, because the arithmetic inside an AI model really is that simple. Multiply a number, add it to another number, do it again. Not clever. Just enormous.

Draw a game and you're colouring two million pixels with the same handful of instructions. Run a model and you're multiplying billions of numbers with those same handful of instructions. To the chip, those are the same job.

That's the answer you'll get everywhere, and it's right. It's also the half that doesn't explain what you watched at the top of this article.

What it actually takes to make one word

So let's do one word. Not in theory, one actual word coming out of an actual model.

A model is a file. Inside that file is a very long list of numbers, and those numbers have a name. They're called weights. That's genuinely all a model is: billions of plain numbers sitting in a file, each one a setting that got nudged into position while the model was being trained. There's nothing in there you could open and read. It's just the numbers.

To produce one word, the machine takes everything you've written so far and multiplies it against those numbers, one after another, all the way through the file.

Not some of them. All of them. Every number in the file, for every single word that comes out.

This is the fact everything else hangs off, so it's worth stating plainly rather than in passing. It comes from the standard analysis of how these models run on hardware (Yuan et al., arXiv:2402.16363). Generating each word requires touching every weight in the model, because every weight participates in the calculation. There's no way to consult part of the file.

The pipe, not the cores

Which means that before it can hand you one word, it has to read the entire model out of memory. Then for the next word, it reads the entire model again. And again, for every word of the answer, start to finish.

Those thousands of cores can do the multiplying almost instantly, which was never in doubt, and the real question was always how fast the numbers could get across to them.

That speed has a name, and it's the most important thing in this article. It's called memory bandwidth, and it means how many gigabytes per second the chip can pull out of memory and deliver into the cores. Think of it as the width of a pipe. That width is fixed, a property of the hardware you bought, and no amount of processing power will ever widen it.

Here's the shape of the problem.

A byte is the basic unit of computer storage, roughly the size of one letter of text. For each byte that comes down that pipe, the cores do about one piece of arithmetic. One.

There's a name for that ratio too. It's called arithmetic intensity, and it means how much maths you get out of each byte you fetch. High arithmetic intensity means you fetch a number once and then do a great deal of work with it, which is the situation any processor wants to be in, and sitting at about one operation per byte is close to the worst case there is.

It's worth knowing where that number comes from, because it's the root of everything else here and it comes back twice before the end. When the machine writes one word, it is multiplying the model's numbers against a single column of values, the one representing the text so far. Each weight gets pulled out of memory, used once, and dropped. Nothing is reused, so the fetch is the whole cost, and that gives an arithmetic intensity of about 1.0 no matter how big the model is. Hold on to the phrase "used once", because the two surprises later in this article are both about what happens when a fetched number gets used more than once.

So the cores finish, and then they sit there. Most of the chip does nothing at all, because the numbers haven't arrived yet.

The cores were never the bottleneck. The pipe is. In the roofline analysis of these systems, word generation sits below the hardware's compute ceiling for very nearly every combination of GPU and model you can name, which is the technical way of saying the same thing.

The arithmetic you can do on the back of an envelope

The useful consequence is that the speed you can get is not a mystery. You can work it out on paper.

bandwidth  /  model size  =  words per second

Take the memory bandwidth and divide it by the size of the model file. That's your ceiling.

Work it at two scales. A model that takes up four gigabytes, on a laptop-class chip carrying something like seventy gigabytes a second, gives you about seventeen words a second. That's a bit over a thousand words a minute, so you watch the text arrive rather than getting it in one go.

Put the identical model on a large data centre card, where the pipe carries thousands of gigabytes a second rather than tens, and the same sum gives you hundreds of words a second. Faster than you could follow.

Same model. Same arithmetic. The same multiplications, in the same order, arriving at the same answer. The only thing that changed is how quickly the numbers could travel.

That sum gives you a ceiling, and nobody quite reaches a ceiling. Published measurements land somewhere around 60 to 85 percent of it. One optimised implementation reports 82 percent of the theoretical bandwidth limit (arXiv:2505.22758), and measurements on smaller devices report 84.5 and 79.2 percent on two different workloads (arXiv:2502.10659). The gap is the software doing the work having costs of its own, and the fact that reading the model isn't quite the only thing going on.

So the number isn't a promise. It's a wall you can't get past, and real machines end up somewhere just underneath it.

One aside, because you'll hit it the moment you measure this yourself. The machine doesn't count in words, it counts in tokens. A token is the chunk of text a model actually reads and writes: usually a common word, a fragment of a longer word, or a piece of punctuation. English averages a bit under a word per token, so the two numbers are close but not identical. Every tool you run will report its speed in tokens per second. The arithmetic above is the same either way, and when I say words and your screen says tokens, we mean the same thing.

Two things this explains

Two things you've very likely noticed and never had a reason for.

The first is that a bigger model is slower, and it's slower almost exactly in proportion. Twice the size, roughly half the speed. That looks strange if you picture the machine thinking harder about a harder problem, and it's completely obvious the second you know it has to drag twice as much down the very same pipe. The measured relationship holds the same way in the other direction: words per second scales nearly linearly with memory bandwidth, which is exactly what you'd expect if bandwidth is the thing setting the pace.

The second is stranger. You can shrink a model and it gets faster, while doing the identical arithmetic.

There's a technique called quantization, and all it does is store each of those weights with less precision. Instead of recording a number to a long string of decimal places, you keep a rougher version of it. The list is the same length. The multiplications are the same multiplications. The file is simply smaller.

And it runs faster, roughly in proportion to how much smaller it got. Nothing about the thinking changed. There was just less to carry.

I want to be careful about how I'm stating that one. It isn't a separate measured finding I'm reporting. It follows from everything above: if the time is set by how many bytes have to move, then moving fewer bytes takes less time. That's the direction the reasoning runs, and it's worth knowing which claims in an article are measurements and which are consequences.

The other half, so the picture is honest

I've been unfair to those thousands of cores, and it's worth putting right.

Reading your question and writing the answer are two completely different jobs, and they have names.

The first is called prefill. When you send a long prompt, the machine can work on all of it at the same time, every word of your input, in parallel, across those thousands of cores at once.

This is the first place that "used once" stops being true, and it's the whole reason prefill behaves differently. Writing a word multiplies the weights against one column of numbers. Reading a thousand-word prompt multiplies the same weights against a thousand columns, and crucially each weight is fetched once and then used a thousand times before it's dropped. The arithmetic intensity that was stuck at 1.0 is suddenly in the hundreds, the pipe stops being the constraint, and the cores finally have more work than they can get through. That part genuinely is heavy arithmetic and it's precisely what a GPU was built for. It flies.

Writing the answer is called decode, and it goes one word, then the next, each one needing the whole model brought across again, and each one held up by the pipe.

Same chip. Same model. One job lights up the entire processor, the other leaves most of it sitting in the dark. This split is well established in the literature on serving these models, and it's the reason systems designed for throughput go to some trouble to overlap the two (arXiv:2308.16369).

It's also why a very long question costs you surprisingly little while a very long answer costs you a lot, and if you have ever pasted several pages into one of these things and been surprised how quickly it came back, that was this.

Where this picture is too simple

I've explained all of this with one pipe and a wall of cores, and I should say plainly that the picture is simplified. There's a great deal of cleverness inside a real chip that I've gone straight past. If you go on to read the actual literature, expect it to be messier than this.

But there's one complication worth having, because it explains something you've felt.

Everything so far assumes the machine is answering you, and only you. One person, one conversation, the whole model dragged across for every word you get back.

Now put a thousand people on that machine at the same moment. The instant those numbers arrive, they can be used for all thousand conversations at once. One trip down the pipe, a thousand times the work out of it.

That's "used once" breaking for the second time, and it's the same trick prefill was pulling. A weight fetched for your conversation is the identical weight the other nine hundred and ninety-nine conversations need, so one trip down the pipe now feeds a thousand calculations instead of one. The ratio that was stuck at about one operation per byte climbs hard, and push it far enough and the cores become the limit again, which is exactly where a GPU wants to be. The same analysis that shows word generation is bandwidth-limited also shows the bottleneck moving back to compute once the batch size grows.

This is called batching, and it's the reason a hosted model can feel quicker than the same model running on your own machine. Your machine has one conversation to work with. Theirs has thousands.

Three sentences, and then the ask

To write one word, the machine reads every number in the model. The pipe carrying those numbers is narrower than the cores are fast. So the size of the model and the width of the pipe set your speed, and raw processing power mostly doesn't.

So are GPUs good at AI? Yes. Thousands of small cores, one tiny piece of arithmetic, repeated forever. That's a real answer and it's why they got the job. They're just not as good as you'd expect, and now you know what to blame.

One thing I'd like from you, and it costs about a minute. Look up the memory bandwidth of whatever you run models on. Divide it by the size of the model file in gigabytes. That's your ceiling. Then run the thing, see what you actually get, and put both numbers in a comment.

I want to know where real machines land. The published figures cluster somewhere between 60 and 85 percent of that ceiling, and every one of them comes from a lab or a paper. Whether the machines people actually have at home fall in the same place is a question none of those papers answer, and about forty of you could.

Next: what all those numbers are actually doing. "Multiply your question by four billion numbers" is a sentence I got through very quickly back there, and it deserves an answer of its own.

🎉 Enjoyed this article? Your support means the world to me!

🎬 Subscribe on YouTube for video versions of these posts: https://www.youtube.com/@swift-pal

💼 Let's connect on LinkedIn for more professional insights: https://www.linkedin.com/in/karan-pal

☕ If this saved you some time, you can buy me a coffee: https://coff.ee/karanpaledx