2 min read AI-generated

Sonnet 5 Tokenizer: Same Price, 40% More Tokens

Copy article as Markdown

Sonnet 5 ships with a new tokenizer. Sounds harmless — but it means the same text now costs up to 40% more tokens. Here's what the numbers actually look like.

Featured image for "Sonnet 5 Tokenizer: Same Price, 40% More Tokens"

Sonnet 5 is here, and the pricing looks attractive at first glance: $2 input, $10 output per million tokens — an introductory rate through the end of August. After that, the standard $3/$15 kicks in, same as Sonnet 4.6. Sounds like ‘better model, same price.’ But there’s a catch.

The New Tokenizer

Sonnet 5 ships with a new tokenizer. Anthropic mentions it almost in passing in the release notes: the same input text produces approximately 1.0 to 1.35x more tokens than Sonnet 4.6.

Simon Willison ran independent tests — and found even higher numbers in some cases:

  • English text: ~1.42x more tokens
  • Spanish text: ~1.33x more tokens
  • Python code: ~1.28x more tokens
  • Simplified Chinese: ~1.01x (essentially the same)

In plain terms: if you used to pay for 1,000 tokens on an English text, you’re now paying for 1,420 — for the exact same content.

What This Means for Costs

At the introductory $2/$10 rate, it roughly evens out. Willison calculated that for English text, you’re effectively paying about $2.84/$14.20 per million ‘real’ tokens. That’s just below the old $3/$15 Sonnet 4.6 rate. So roughly cost-neutral — with a better model.

But come September, the standard $3/$15 applies. At that point, the same English text will effectively cost ~$4.26/$21.30 per million tokens. That’s about 40% more than Sonnet 4.6. For code-heavy workloads, it’s still around 28% more.

If you primarily work in Chinese, you’ll barely notice a difference. But for most Western languages and code, this is a real price increase — wrapped in a technical change.

Why the New Tokenizer?

Anthropic doesn’t explain the change in detail. New tokenizers can improve model quality by capturing finer language structures. That’s a legitimate technical trade-off. But the communication is misleading when the list price stays the same and the tokenizer effect is buried in the fine print.

My Take

The introductory phase is fair — through August you’re paying less or the same as before for a better model. But what happens in September will be telling. If Anthropic keeps the standard price at $3/$15 while the tokenizer inflates token counts by 30-40%, that’s effectively a price increase. Whether you call it ‘hidden’ or ‘technically necessary’ is a matter of perspective. Finout has a really detailed breakdown of this, by the way — worth a read.


Sources: Simon Willison: What’s new in Claude Sonnet 5 · Finout: Claude Sonnet 5 Pricing 2026 · Anthropic: Claude Sonnet 5