Tokenizer
The component that splits text into tokens — and the reason two models with the same per-token price can cost different amounts.
A tokenizer is the algorithm that breaks text into the tokens a model is billed on. Different model families use different tokenizers, so the same paragraph can become a different number of tokens depending on which model reads it. That is why per-token price is not always an apples-to-apples comparison: a model with a slightly higher rate but a more efficient tokenizer can be cheaper for your text than a lower-rate model that splits the same text into more tokens.
The effect is real and sometimes large. A newer tokenizer generation can produce noticeably more tokens for the same text — this explorer flags, for example, that some recent Claude models use a tokenizer that yields roughly 30% more tokens per unit of text. When that is the case, the headline per-1M rate understates the true cost-per-task relative to an older tokenizer.
The practical lesson: compare cost-per-task, not just cost-per-token, especially across model generations and vendors. The only reliable way is to run representative text through each model’s tokenizer and measure the actual token counts, then apply the rates. Where a tokenizer skew is known, this explorer surfaces it as a note on the model and compare pages so the comparison is honest rather than misleading.
FAQ
Why do two models with the same price cost different amounts?
Because their tokenizers split text differently. A model that turns your text into more tokens costs more even at the same per-token rate.
How do I compare models fairly across tokenizers?
Compare cost per task, not per token: run your actual text through each model’s tokenizer, measure the token counts, then apply each model’s rates.