Your CPU Has More Cores Than Your Job Can Use

You have seen both of these, on the same machine, probably in the same afternoon.

One is the export. You hit render on a video, open Activity Monitor or Task Manager, and every bar is up. Eight cores, all of them busy, the fan going hard. It feels like the machine you paid for.

The other one is the annoying one. One bar is pinned at 100%. The other seven are flat. The fan is just as loud, the app is frozen, and seven eighths of your processor is sitting there doing nothing about it.

Bar three, not any of the other seven — the picture the whole piece is explaining.

Same machine. Same operating system. Same afternoon. The thing that decides which of the two you get is not the machine and not the operating system, and once you can see it you can predict it from the job description alone, before anything starts.

What a core is actually handed

A core does not run a program. It runs a thread.

A thread is one list of steps, in order, where each step starts where the last one finished, and that is the whole of it. It is the unit the operating system hands out: not “Photoshop”, not “the export”, but one specific list of instructions with a place in it where the core currently is. A program can own several of these lists at once, and most do. Each one is still a separate list.

The arrows are the whole argument: there is no seam in the middle of a hundred percent for a second core to hold on to.

This word is the reason the other seven cores cannot help you. One list is read by one core. Step two begins exactly where step one ended, so there is only one place in that list where reading is happening, and only one core can be standing at it.

Follow the core along the list. It takes a step, finishes it, takes the next one. A hundred percent is what that looks like from outside. It means that core is getting through its list as fast as a core can get through a list. That is a reading of full effort, and almost everybody takes it as a reading of failure. Activity Monitor has no column for how much of the list is left.

The one move the operating system has

The obvious fix is to give the job to a different core, and that move is real. The operating system can pick up the whole list and put it somewhere else. What arrives on the other core is the same steps, in the same order, taking the same time, in a different row of the chart.

Same steps, same order, same time — the only thing that changed is which row of the chart it is drawn on.

Moving a list and splitting a list are different operations. Only one of them exists.

Plenty of programs do run several lists at once, and when they do you see several bars go up. That is real and it is common. It does not help the job you are currently sitting here waiting for, because several lists is several lists. None of them is half of the one list that is holding you up.

Fast cores and slow cores are the other thing people raise here. Some chips have both, and the operating system does choose between them, and moving your list onto a fast core does finish it sooner than a slow one would. It changes which core the list runs on. It never changes how many.

Why the work is one list

The list is one list because each step needs the answer the step before it produced. Hook the steps together and there is nothing left to argue about: step two takes step one's output as its input.

The hook is the picture of the dependency — step two cannot start until step one's hook is there to catch it.

Plenty of ordinary work is built that way, and once you look for it you find it everywhere. A running total cannot add the fifth number until the first four have been added, because the thing it is adding to is the result of the fourth step. A physics simulation works out where everything is in this frame from where everything was in the last one, so frame two hundred is two hundred steps deep in a queue that only goes one way.

The clearest example running on ordinary machines right now is a model writing text one word at a time. If you have run one locally, you have watched it happen. A model here is a program that produces text by producing one piece, then the next, then the next, and the part that matters for us is where each new piece comes from. It comes from everything already written, fed back in.

That is not a guess about a particular product. It is the defining property of the design, stated in the 2017 paper that introduced the architecture these models are built on: “At each step the model is auto-regressive, consuming the previously generated symbols as additional input when generating the next” (Attention Is All You Need, section 3). Auto-regressive means a thing whose own output becomes its next input. The loop is the point. Take the feedback away and the model has no idea what it already said, so the fifth word would be produced as if the first four had never happened.

Two words in that quote are doing careful work, and one of them is symbols. The pieces a model produces are not exactly words. For the shape of the work that makes no difference at all, so I am going to keep saying words.

The consequence is the whole article. The second word cannot begin until the first word exists. Speed has nothing to do with it. The input to the second step is the output of the first.

So put the second word on another core

Nothing stops you. Go ahead and put it on core four.

It lands there. Then it sits there, waiting for a first word that does not exist yet. When the answer finally arrives it starts, and it finishes at exactly the moment it would have finished if you had never moved it. Two cores busy now, doing what one core was doing, and the clock has not noticed.

It lands there and then it sits there — a second core, doing nothing, because the word it needs has not been produced yet.

You can move the work. You can pay for more cores. You cannot produce an answer before the thing it depends on.

Work that is not shaped like that

Take a thousand photos and resize them.

Photo 916 does not need anything that photo 915 worked out. Nothing gets passed forward. Nothing hooks to anything. The order they run in does not matter, and any one of them can start the moment there is somewhere to start it.

The hook that connected every other shot in this piece is drawn broken on purpose — there is nothing for it to catch.

So deal them out. Eight photos to eight cores, all eight bars up together, the queue draining. That is your export. Same operating system, same eight cores, same afternoon as the frozen app. The only thing that changed is what was being asked of them.

Every row moving at once is what an export looks like, from the same operating system that pinned one bar a minute ago.

Those are the two shapes, and that is the whole answer to the difference at the top. What decides which one you are looking at is the work itself, not the operating system and not the number of cores on your invoice. Pieces that do not need each other's answers spread across as many cores as you own, and a chain of steps that feed each other does not.

Watch on YouTubeWhy Your CPU Uses Only ONE CoreThe same question as a video, opening on the frozen app and the seven idle cores, and working out what the operating system is actually unable to do.

Real jobs are both at once

Almost nothing you run is purely one shape. A real job has a chain running through it and a pile of independent work hanging off the chain, and the interesting question stops being which shape is this and becomes how much of it is chain.

So take one job. Twenty minutes on a single core.

Say a tenth of it is a chain and the other nine tenths splits up cleanly. I want to be blunt about that tenth, because it is the number a reader is most likely to walk away quoting. It is an assumption. I picked it because it makes every figure below come out whole. I have no measurement of the chain's share of a typical program, nobody has one for your program, and nothing in the arithmetic that follows depends on the tenth being true. Change it and every number changes with it. The behaviour does not.

A tenth chain, nine tenths splittable — the assumption the rest of the arithmetic is built on, and it is only that.

A tenth of twenty minutes is two minutes of chain. The other eighteen minutes can be spread.

Give it eight cores. The eighteen minutes splits eight ways and comes back as two and a quarter. So the job finishes in four minutes and fifteen seconds instead of twenty. Call that 4.71 times faster.

Now give it more.

one 20-minute job, a tenth of it a chain
  cores    finishes in      faster by
      1    20 min  0.0 s        1.00x
      8     4 min 15.0 s        4.71x
     16     3 min  7.5 s        6.40x
     32     2 min 33.8 s        7.80x
     64     2 min 16.9 s        8.77x
   1000     2 min  1.1 s        9.91x

Eight cores took the eighteen splittable minutes down to two and a quarter — the two-minute chain never moved.

Read down the middle column rather than the right one. Going from eight cores to sixteen saves you 67.5 seconds. Sixteen to thirty-two saves 33.75. Thirty-two to sixty-four saves 16.875. Every doubling of the entire machine buys exactly half of what the last doubling bought, and it is heading somewhere specific.

It is heading to two minutes. Not close to two minutes. Two minutes, and not one second better, at any core count that will ever be built, because two minutes is how long the chain is. The chain was never eight things. It was always one thing after another, and there is nothing in a core count that has anything to offer it.

That ceiling has a name and a date. It is Amdahl's law, written down in 1967, and it is this:

S = 1 / ( (1 - p) + p/N )

S is how many times faster you finish. N is how many cores you throw at it. p is the share of the job that can be spread out, so 1 - p is the share that has to happen in order, which is the part I have been calling the chain. It has a proper name too, the serial fraction, and that is the quantity the whole argument turns on. Put in p = 0.9 and N = 8 and you get 4.71. Put in an N of infinity and the p/N term vanishes, leaving 1/0.1, which is 10.

Ten times faster. That is what every core that will ever exist is worth on this job. Seven cores you already own take it from twenty minutes to four fifteen; the entire future of the industry takes it from four fifteen to two.

Seven cores you already own get you most of the way there; the number in the corner is the price of the rest.

The check I thought I had done

I worked that 4.71 out twice. Once the slow way, in minutes, and once by putting p = 0.9 and N = 8 into the formula. Both gave 4.71, and I wrote in my notes that the number was checked two independent ways.

It is not two ways. It is one way, run twice.

Watch what the minutes version actually computes. The job is T minutes on one core. The chain part is T(1-p), the spreadable part is Tp divided across N cores, and the speedup is the original time over the new time:

S = T / ( T(1-p) + Tp/N )

Every term underneath has a T in it, so the T cancels, and what is left is 1 / ((1-p) + p/N), which is the identity, character for character. The reason Amdahl's law never mentions how long your job takes is that the length cancels out of it.

That cancellation is exactly what made the second route look independent. It carried a quantity the first one did not have, twenty minutes, and the quantity it carried was the one that disappears. A second derivation with the same assumption inside it agrees with the first by construction. It is still worth running, because it catches a slip in the arithmetic, and it corroborates nothing whatsoever about whether the reasoning underneath it is right.

I now treat “I got the same answer two ways” as a claim I have to check rather than a reason to stop checking. The test is quick: if one route uses a quantity the other one does not, find out whether that quantity survives to the end. If it cancels, you ran the same derivation twice.

Both routes give 4.71 because both routes secretly assume p equals 0.9 — the T just proves the arithmetic, not the assumption.

The only lever in the whole thing

The ceiling belongs to the chain. So shorten the chain.

Take the same twenty minute job and get the chain down to one minute instead of two, and the floor moves with it: the best case is now one minute flat, which is twenty times faster instead of ten. The ceiling doubled and nobody bought a single core.

Halving the chain doubles the ceiling. The core count never moved.

I would have got the next part wrong if I had not done the arithmetic. It is the reason this section exists. On eight cores, halving the chain barely helps.

same job, eight cores
  chain 2 min    4 min 15.0 s     4.71x
  chain 1 min    3 min 22.5 s     5.93x

Fifty-two and a half seconds. You doubled the ceiling from ten to twenty and collected 26% of a speedup, because on eight cores you were never anywhere near the ceiling to begin with. Most of what shortening the chain buys is only collectable by a machine with a great many cores. At sixteen cores the same change is worth 9.14 times instead of 6.40, and the gap keeps widening from there.

The seven idle bars and the two-minute floor are not two facts. They are one fact, seen at two zoom levels. Each step needs the answer the step before it produced, so at the small scale nothing can start on another core, and at the large scale the finish line stops moving. The idle cores are the floor, drawn from close up.

The other law, and why it does not help here

There is a second law next to Amdahl's, Gustafson's, and it says something genuinely different: in practice people do not hold the job still and add cores, they buy a bigger machine and then ask it a bigger question. Render at higher resolution. Resize ten thousand photos instead of a thousand.

The thousand photos above were already that regime, without the name. Make it ten thousand and all eight cores stay full, because the extra work is more independent pieces and independent pieces are exactly what spreads.

It does not touch the job in front of you. Gustafson's law is about what to do with a machine, not about what to do with an afternoon. Your job's size was fixed before the machine ever saw it, and nothing in that law is about finishing a fixed job sooner.

The picture that does not move is the point — a bigger job fills the same eight cores, it does not shrink this one.

Back to the two pictures

The export lights up every core because a thousand photos have a thousand starting points. The frozen app pins one core because the job you gave it has exactly one, and everything after it is waiting on the answer before it.

Nothing has gone wrong in the second picture, and the operating system did the only thing available to it, which was to pick a core and hand over the list. The core it handed the work to is going flat out. The seven that look wasted have nothing in front of them that does not depend on it. What you are looking at is not a machine wasting seven eighths of itself. It is the shape of your job, drawn accurately, on a chart that was only ever showing you the truth.

Not a machine wasting itself — a job, drawn accurately, on a chart that was only ever telling the truth.

That moves the cause somewhere useful. It stops being a question about your machine or your invoice and becomes a question about the work: how much of this has to happen in order? That is answerable, sometimes changeable, and it is the only number in the entire piece that moves the finish line.

So the next time one bar is pinned and seven are flat, read it as the shape of the job rather than a fault. That shape was decided before the machine ever saw it.

One gap I cannot close on my own. The tenth is an assumption, and the honest position is that I have no measured chain share for any real program, so the ladder above tells you the behaviour and not your numbers. If you have ever watched a job stop getting faster as you gave it more cores, tell me what the job was and roughly where it stopped paying, either a core count or the point where doubling the machine stopped showing up in the clock. Enough of those and the tenth stops being a number I made up to keep the arithmetic round.

🎉 Enjoyed this article? Your support means the world to me!

🎬 Subscribe on YouTube for video versions of these posts: https://www.youtube.com/@swift-pal

💼 Let's connect on LinkedIn for more professional insights: https://www.linkedin.com/in/karan-pal

☕ If this saved you some time, you can buy me a coffee: https://coff.ee/karanpaledx