An Empty String Costs 8 Tokens: What Apple's tokenCount(for:) Actually Measures

The first version of my token counter told me the word test was 9 tokens.

GPT-4o says 1. So does GPT-3.5. Four characters, one ordinary English word, one token.

Nine is not a rounding difference or some tokenizer quirk you shrug at. It is wrong by a factor of nine, and it is wrong in the one direction that will actually hurt you when the window you are budgeting against holds 4,096 tokens and not a single one more.

The tool is called Token Budget, it is open source, and you can run everything below yourself. The bug turned out to be a misreading of Apple's API that I suspect a lot of people are about to make, because the name invites it.

tokenCount(for:) is not text.count for tokens

The smallest program that shows it needs macOS 26.4 or later, since that is where tokenCount(for:) landed.

// TokenProbe.swift
//
// Run it with:
//   xcrun swift -swift-version 6 -target arm64-apple-macos26.4 TokenProbe.swift

import Foundation
import FoundationModels

let model = SystemLanguageModel.default

guard case .available = model.availability else {
    print("Apple Intelligence is not available here: \(model.availability)")
    exit(1)
}

let samples = ["", "a", "test", "test test", "hello world"]

print("contextSize = \(model.contextSize)")
print("")
print("input           chars  tokenCount")

for text in samples {
    let count = try await model.tokenCount(for: text)
    let quoted = "\"\(text)\""
    print(quoted.padding(toLength: 15, withPad: " ", startingAt: 0)
        + " \(text.count)".padding(toLength: 6, withPad: " ", startingAt: 0)
        + " \(count)")
}

One note before you run it, because it cost me a minute. xcrun swift TokenProbe.swift on its own refuses to compile, complaining that 'tokenCount(for:)' is only available in macOS 26.4 or newer, on a machine that is quite happily running something newer than that. Script mode picks a conservative deployment target of its own accord, so you have to pass -target explicitly.

That is why the run command sits in the header comment.

The shape of the probe matters more than it looks. The samples climb from nothing upward, empty string then one character then one word then two, and the ladder is the point. A constant error disappears into long text and screams on short text. A proportional error does the reverse. One measurement cannot tell you which kind you are looking at, and picking the wrong one to test on is how this bug survived in the first place.

The availability guard is not politeness either. With Apple Intelligence switched off, tokenCount(for:) does not return zero or some sentinel, it throws, and the error you get is a ModelManagerError about assets rather than anything mentioning Apple Intelligence. Without the guard the script dies looking like a broken toolchain when the actual problem is a settings toggle.

The output:

contextSize = 4096

input           chars  tokenCount
""              0     8
"a"             1     9
"test"          4     9
"test test"     9     10
"hello world"   11    10

An empty string is 8 tokens.

That one line explains the rest of the table. tokenCount(for:) never measures a string on its own; it measures what that string will cost as a prompt, and a prompt is your text wrapped in whatever chat scaffolding the model expects around it, which means role markers, turn boundaries, and the structural tokens that tell the model a user is speaking and where the speaking stops.

Subtract the 8 and everything lines up. test is 1 content token. test test is 2. hello world is 2. That agrees with every other tokenizer.

Worth confirming rather than assuming, so: 300 repetitions of word come back as 309, and repeated calls on the same string return the same number every time. A fixed 8, whatever you hand it.

Fixing it without hardcoding 8

The obvious repair is to subtract 8.

I did not want to write 8 into the source, though. It is not my number. It belongs to a model Apple ships and can quietly reformat in a point release without telling anybody, at which point a hardcoded 8 stops being a fix and becomes a silent off-by-some that nobody goes looking for.

So the tool measures it at startup:

// AppleTokenCounter.swift

/// A token count split into the part that is your text and the part that isn't.
public struct Count: Sendable {
    /// Tokens the text itself contributes.
    public let content: Int
    /// Fixed scaffolding charged on top, whatever the text.
    public let overhead: Int
    /// What the model actually spends from its window.
    public var total: Int { content + overhead }
}

public static func count(in text: String) async throws -> Count {
    let model = SystemLanguageModel.default
    let overhead = try await model.tokenCount(for: "")
    let total = try await model.tokenCount(for: text)
    return Count(content: max(0, total - overhead), overhead: overhead)
}

Two tokenCount calls instead of one, and the number stays current. If Apple's prompt format changes, the empty-string probe changes with it and nothing downstream needs touching.

I keep the two numbers apart rather than quietly reporting the corrected one, because they answer different questions. Comparing Apple's tokenizer against OpenAI's calls for the content count, since that is the like-for-like measure. Asking whether your prompt fits calls for the total, since the window is spent either way.

Then I checked the number against reality, and it was still wrong

With the counting fixed I wanted to watch it fail, so I fed the tool a 47,000-character article and let it run straight past the 4,096-token window.

tokenCount(for:) reported 12,649 for that article. Then the session threw this:

exceededContextWindowSize: Content contains
12663 tokens, which exceeds the maximum
allowed context size of 4096.

12,663 against 12,649.

Fourteen tokens the counting API knows nothing about. Small enough to wave off, and exactly the kind of thing that bites at the boundary, so it was worth pinning down properly. Conveniently the session reports its own accounting inside the overflow error, which means you can measure the gap by overflowing on purpose:

// SessionCost.swift
//
// Compares what tokenCount(for:) reports against what a session actually
// charges, by deliberately overflowing the context window and reading the
// number out of the error.
//
// Run it with:
//   xcrun swift -swift-version 6 -target arm64-apple-macos26.4 SessionCost.swift

import Foundation
import FoundationModels

let model = SystemLanguageModel.default

guard case .available = model.availability else {
    print("Apple Intelligence is not available here: \(model.availability)")
    exit(1)
}

/// The session's own accounting, recovered from the overflow error.
/// Returns nil when the prompt fits, because then there is no error to read.
func chargedTokens(for prompt: String, instructions: String?) async -> Int? {
    let session = instructions.map { LanguageModelSession(instructions: $0) }
        ?? LanguageModelSession()
    do {
        _ = try await session.respond(
            to: prompt,
            options: GenerationOptions(maximumResponseTokens: 4)
        )
        return nil
    } catch let error as LanguageModelSession.GenerationError {
        let description = "\(error)"
        let pattern = #"contains (\d+) tokens"#
        guard let match = description.range(of: pattern, options: .regularExpression) else {
            return nil
        }
        return Int(description[match].filter(\.isNumber))
    } catch {
        return nil
    }
}

// 5,000 words is comfortably past the 4,096 window, so the error always fires.
let prompt = String(repeating: "word ", count: 5_000)
let promptCount = try await model.tokenCount(for: prompt)
let emptyCount = try await model.tokenCount(for: "")

print("tokenCount(for: \"\")     = \(emptyCount)")
print("tokenCount(for: prompt) = \(promptCount)")
print("")

let instructionSets: [(label: String, value: String?)] = [
    ("none", nil),
    ("short", "You are a helpful assistant."),
    ("long", String(repeating: "rule ", count: 200)),
]

print("instructions  charged  charged - tokenCount")

for (label, instructions) in instructionSets {
    guard let charged = await chargedTokens(for: prompt, instructions: instructions) else {
        print("\(label): prompt fitted, nothing to read")
        continue
    }
    print(label.padding(toLength: 14, withPad: " ", startingAt: 0)
        + "\(charged)".padding(toLength: 9, withPad: " ", startingAt: 0)
        + "\(charged - promptCount)")
}

Three things in there deserve a word, because none of them are obvious and one of them is frankly unpleasant.

It reads a number out of an error message with a regex. That is not a design choice I enjoyed. Nothing in the API will tell you what a session actually charged for a prompt; the only place that number is ever stated is inside the text of the overflow error. So the only way to see the session's own accounting is to overflow on purpose and parse the complaint. It is string-matching against a message Apple can rewrite in any release, and if they do, this returns nil and the script quietly reports nothing rather than lying. That is the best failure mode available here, not a good one.

`maximumResponseTokens: 4` keeps the response out of the way. Without it the model is free to plan a long answer, and then you are no longer measuring what the prompt costs, you are measuring prompt plus whatever the model reserved for itself. Four is small enough to be noise and large enough to be legal.

Returning nil when the prompt fits is the contract. No overflow means no error, and no error means no number to read, so the caller has to handle "it fitted" as a distinct outcome rather than as zero. Treating it as zero would silently report a 0-token charge for every prompt small enough to succeed, which is exactly the sort of plausible wrong answer this whole article is about.

The three instruction sets are there to separate a fixed cost from a proportional one. None, short, and a deliberately long one at 200 repetitions. If the gap were proportional it would grow across those rows. It does not.

Output:

tokenCount(for: "")     = 8
tokenCount(for: prompt) = 5009

instructions  charged  charged - tokenCount
none          5023     14
short         5029     20
long          5224     215

The gap is 14 with no instructions, at 5,009 tokens and at 12,649 tokens alike. Constant, not proportional.

The instructions rows show where it comes from. tokenCount says the short instruction string is 14 tokens, which is 6 of content plus the familiar 8; the long one is 209, so 201 of content. Line those up against what the session actually charged and the whole thing collapses into a single equation:

charged = content(prompt)
        + content(instructions)
        + 22

Check that against all four measurements and it holds at 22 every time, including the 47,000-character article that started this.

So the session's real fixed cost is 22 tokens, and tokenCount(for:) reports 8. Two different overheads, and the one you can measure directly is not the one you are billed for.

Budget with tokenCount(for:) alone and you are under by 14. Get clever and add tokenCount(prompt) + tokenCount(instructions) and you are under by 6, having counted the 8-token scaffolding twice and still missed the session's 22.

The window has a decoy in it

One more thing before you lean on any of these numbers.

SystemLanguageModel.contextSize reads 4096, which happens to be the real window. It is also @backDeployed(before: macOS 26.4) with a hardcoded 4096 fallback body, so on 26.0 through 26.3 the property hands back that literal without consulting the model at all.

The placeholder and the truth are the same number.

You cannot tell from the value which one you got, and a test asserting contextSize == 4096 passes cheerfully on an OS where the property is not reading anything.

What I do now

Measure the empty string on startup and subtract it, so the counts stay comparable across vendors. Treat tokenCount(for:) as a lower bound rather than a budget. Leave real headroom, because 22 tokens is nothing at all until your prompt reaches 4,080, at which point it is the difference between a response and a thrown error.

And feed any counting API something degenerate before you trust it.

The 8-token overhead was invisible on a 47,000-character article, an error of 0.06% that nobody would ever catch. On one word it was 9x. Empty strings and single characters are where this class of bug is loudest, which makes them the cheapest place to go looking.

Measured with Swift 6.3.3 against the FoundationModels framework on macOS 26.4+.