There Is No Such Thing as a Token Count

Ask how many tokens a piece of text is and someone will hand you a number.

That number came from whichever tokenizer they happened to have installed. It is right for one model and wrong for every other one, and the gap between them is a great deal wider than the rounding-error framing everybody reaches for would suggest.

Token count only looks like a property of text, the way character count is. Really it belongs to a pair: your text, and one specific tokenizer.

So I built something to measure the disagreement. It takes text and prints what that text costs against each model's context window, using each vendor's real tokenizer rather than a shared approximation. It is called Token Budget and it is open source, so every number below is one you can reproduce.

The same greeting, three tokenizers

Eight phrases, all roughly "hello world", plus two emoji. Counts are content tokens only, with prompt scaffolding stripped out so the columns compare like with like.

language  apple o200k cl100k  sample
English       2     2      2  Hello world
Hindi         3     5     13  नमस्ते दुनिया
Japanese      2     2      4  こんにちは世界
Arabic        6     4     10  مرحبا بالعالم
Russian       5     4      6  Привет, мир
Chinese       3     3      6  你好，世界
Korean        3     3      9  안녕하세요 세계
Emoji         8     4      6  👋🌍

English is boring. Every tokenizer charges 2 tokens for Hello world, because English is the language all of them were tuned on and there is simply no headroom left in which one of them could pull ahead of another.

The rest of that table is a mess.

cl100k_base is the encoding behind GPT-3.5 and the GPT-4 generation, and it is dramatically worse at non-English than the encoding that eventually replaced it. Hindi: 13 tokens against 5. Korean: 9 against 3. Arabic: 10 against 4. Two encodings from the same vendor, one generation apart, and a Korean sentence written in either of them differs by a factor of three depending on nothing but which one your library happens to load.

If you are serving non-English users on an older OpenAI encoding, that gap shows up on your invoice every month, and it lands hardest on the users whose language the vocabulary covered worst.

The ranking flips depending on what you feed it

Apple's on-device tokenizer is the interesting column. It does not sort cleanly against either OpenAI encoding.

It beats o200k_base on Devanagari, 3 tokens against 5. Ties on Japanese, Chinese and Korean. Loses on Arabic, 6 against 4, and on Russian, 5 against 4.

And it is terrible at emoji.

👋🌍 costs 8 tokens on Apple's tokenizer against 4 on o200k_base. There is a clean reason for that: those two emoji are 8 bytes of UTF-8, and Apple charges exactly one token per byte. A single 👋 is 4 bytes, so 4 tokens. The vocabulary holds no merges for these characters at all, so the tokenizer falls all the way back to raw bytes, which is what byte-level BPE does when it has nothing better to reach for.

Now compare Devanagari. नमस्ते दुनिया is 37 bytes of UTF-8, and Apple squeezes it into 3 tokens. Same tokenizer, same afternoon, and the compression ratio swings from 12:1 to 1:1 depending on nothing more than whether the vocabulary happens to cover that corner of Unicode.

Long-form English behaves differently again. A 47,000-character technical article, mostly prose with Swift code blocks:

Apple on-device   12,641 tokens
GPT-4o (o200k)    10,866 tokens
GPT-3.5 (cl100k)  10,812 tokens

Apple needs about 16% more tokens than either OpenAI encoding for identical English text.

So the tokenizer that wins on Hindi loses on English, on emoji and on Arabic. No overall ranking falls out of this. What you have is a vocabulary, and the only question worth asking is how well it happens to cover the text you are about to feed it.

The vendor that will not tell you

There is a third case, and most token counters get it quietly wrong.

Anthropic publishes no offline tokenizer for Claude. No rank table to download. No library that reproduces the vocabulary locally. An exact count means a network call to POST /v1/messages/count_tokens, every time.

Most tools deal with that by running the text through tiktoken and printing whatever comes out, because it is sitting right there and the number looks plausible. Anthropic's own documentation is blunt about the result: tiktoken undercounts Claude by roughly 15 to 20% on ordinary prose, and by considerably more on code or non-English input.

Look back at the first table and that stops being surprising. o200k_base and cl100k_base are two encodings from the same vendor, one generation apart, and they disagree by 2.6x on Hindi. A third vendor's tokenizer gives you a confident answer to a question you did not ask.

My tool prints nothing in that column. Model, context window, and a line saying why the count is missing.

The window matters more than the tokenizer

While I was comparing counts I nearly walked past the bigger number on the screen.

Apple's on-device model has a context window of 4,096 tokens. GPT-4o has 128,000. The current Claude models sit at 1,000,000.

That 47,000-character article fills 8.49% of GPT-4o's window. Against the on-device model it comes to 308%, and the session throws exceededContextWindowSize rather than truncating politely.

A 16% tokenizer difference is worth knowing about. A 31x window difference decides what you are allowed to build at all, because every technique that quietly assumes you can hand a whole document to the model has to be swapped out for chunking, summarising or retrieval before you write a line of feature code.

One more thing worth knowing. contextSize is @backDeployed(before: macOS 26.4) with a hardcoded 4096 fallback, and the real window on newer systems is also 4,096. Reading that value tells you nothing about which one you got.

One trap if you implement BPE yourself

I wrote the OpenAI side from the published rank tables rather than shelling out to Python, which meant implementing byte-level BPE in Swift.

It degrades silently. That is the trap.

Byte-level BPE starts by mapping every input byte to a single-byte token, then repeatedly merges the highest-ranked adjacent pair. If your rank table is missing even one of the 256 single-byte tokens, the merge loop finds no entry for that byte and drops it on the floor. No error. No warning. Just a count that runs quietly low, and only for the inputs that happen to contain that byte.

Loading now validates all 256 and throws if any are missing.

The wider version of that lesson is why I did not trust my own implementation for a second. Merge order is not something you can eyeball, and a wrong BPE returns believable numbers instead of crashing. So the Swift version is checked against Python tiktoken 0.13.0 with a script that emits reference counts for whitespace runs, repeated newlines, emoji, Devanagari and source code, and those exact numbers are pinned as test expectations.

What to actually do

Count with the tokenizer belonging to the model you are calling. Every time. Cross-vendor estimates are not conservative, they are wrong in a direction you cannot predict.

If you serve non-English users, measure your real traffic instead of a handful of English test strings. The gaps above are wide enough to move a hosting bill, and they land unevenly across your user base.

And if you are targeting on-device, start from the window rather than the tokenizer. Four thousand tokens is roughly one long article. That is the constraint everything else has to bend around.

Counts measured with Swift 6.3.3 against the FoundationModels framework on macOS 26.4+, and against OpenAI's published `o200k_base` and `cl100k_base` rank tables, cross-checked with Python `tiktoken` 0.13.0.