Five Tokenizers Count the Same Chinese Paragraph
Last month I costed out a content platform's LLM budget. I ran twenty thousand Chinese articles through tiktoken's cl100k_base, multiplied, and filed a number. The first month's invoice came back at sixty percent of it.
My first guess was that volume had dropped. The logs said otherwise: request count flat, no retry storm, cache hit rate normal. What was smaller was the token count inside each request. The same Chinese paragraph becomes a different number of pieces depending on which model reads it, and billing follows the pieces.
That is not a bug. It is a vocabulary difference. In an earlier post I measured how many tokens one Chinese character becomes inside tiktoken (Tokenizer Test: How Many Tokens Is One Chinese Character), but that stayed inside one library. This post goes one level down: I downloaded the tokenizer.json from the weight repositories of Qwen, DeepSeek, Llama and Mistral, put them next to OpenAI's two encodings, and counted the same text on all six.
TL;DR: the same 118-character Chinese paragraph costs between 74 and 128 tokens depending on the tokenizer, a 73.0% spread. The same 60-word English paragraph spreads by 4.4%. Vocabulary size does not explain it: Llama-3.1 has 128,256 entries and DeepSeek has 128,815, yet they land 33.8% apart on Chinese. If you need a budget number, count with the tokenizer that belongs to the model you are actually calling.
Where the six numbers come from
Everything ran locally. No API calls, no network at scoring time. tiktoken 0.14.0 supplies the two OpenAI encodings; the other four are loaded from tokenizer.json with Tokenizer.from_file() from tokenizers 0.23.3.
| Tokenizer | Vocabulary | File size | Source |
|---|---|---|---|
| cl100k_base | 100,277 | built in | tiktoken 0.14.0 |
| o200k_base | 200,019 | built in | tiktoken 0.14.0 |
| Qwen3-8B | 151,669 | 10.89 MB | HF Qwen/Qwen3-8B |
| DeepSeek-V4 | 129,280 | 6.37 MB | vendor offline bundle |
| Llama-3.1-8B | 128,256 | 8.66 MB | HF mirror repository |
| Mistral-7B-v0.1 | 32,000 | 1.71 MB | HF mistralai/Mistral-7B-v0.1 |
One detail matters before anything else. Llama-3.1's official repository is gated: requesting its tokenizer.json returns 401. I used an ungated mirror that ships the same weights and the same vocabulary, and I have marked that row accordingly. Which is itself the point: "the official tokenizer" is not a statement you can act on until you know which copy you are holding.
Chinese differs by 73%, English by 4.4%
I took a 118-character Chinese paragraph — about this subject, written the way a normal technical post is written — and counted it six ways:
| Tokenizer | Tokens | Tokens per character | Tokens per 10k characters |
|---|---|---|---|
| cl100k_base | 128 | 1.085 | 10,847 |
| Mistral-7B | 119 | 1.008 | 10,085 |
| Llama-3.1 | 99 | 0.839 | 8,390 |
| o200k_base | 96 | 0.814 | 8,136 |
| Qwen3 | 81 | 0.686 | 6,864 |
| DeepSeek-V4 | 74 | 0.627 | 6,271 |
The high and low differ by 73.0% (10,847 ÷ 6,271 = 1.73). The same experiment on a 60-word English paragraph gives 68, 68, 68, 69, 68 and 71 — a 4.4% spread.
So the pattern is simple: tokenizer divergence is a CJK problem, not a general one. Swapping vocabularies on English load costs you almost nothing; on Chinese load it moves your bill by seventy percent. Any "roughly how many tokens is this" rule of thumb fails across vendors the moment the text is Chinese.
The measurement also lets me check DeepSeek's own conversion note. Their token usage page states, verbatim, "1 English character ≈ 0.3 token. 1 Chinese character ≈ 0.6 token." My paragraph comes out at 74 tokens for 118 characters, or 0.627 per character — 4.5% above their coefficient. The same page also says the actual count "is based on the model's return", which is the honest version: the coefficient is for sanity checks, not invoices.
Vocabulary size does not explain the 73%
My first hypothesis was that a bigger vocabulary means cheaper Chinese. The data refuses it.
Llama-3.1 ships 128,256 entries and DeepSeek 128,815 — a 559-entry gap, effectively the same size. On the same Chinese paragraph Llama needs 0.839 tokens per character and DeepSeek needs 0.627, a 33.8% difference. In the other direction, Qwen3 has 151,669 entries, 17.7% more than DeepSeek, and still spends more per Chinese character (0.686 against 0.627).
What actually varies is Chinese coverage inside the vocabulary. A tokenizer trained on a corpus with heavy Chinese will hold whole words like 产品说明 or 预算 as single entries. A tokenizer trained mostly on English has to fall back toward bytes, and one character becomes two or three pieces. Entry count is the ceiling; the training mix decides how much Chinese fits under it.
Mistral-7B is the clearest case: 32,000 entries, 1.008 tokens per character, roughly cl100k territory and 60.8% more expensive than DeepSeek on Chinese. It is not broken. It was never trained for this.
Official bundle vs Hugging Face mirror: 465 entries apart, 0.0% apart
DeepSeek's token usage page links an offline tokenizer bundle. I downloaded it and diffed it against the DeepSeek-V3 directory on Hugging Face:
- Official bundle: 129,280 entries, tokenizer.json at 6.37 MB
- HF mirror: 128,815 entries, tokenizer.json at 7.48 MB
That looks like two different artifacts: 465 entries and 1.1 MB apart. On the same Chinese paragraph they produce 74 and 74. On the English paragraph, 69 and 69. A 0.0% difference. The extra 465 entries are special-token slots and added entries that never participate in ordinary text segmentation.
That is both reassuring and a trap. Reassuring: on this model, substituting the mirror for the vendor bundle is safe. The trap: "different size, different entry count, identical output" is reported nowhere. You only find out by counting. Change the model version and that 0.0% could become 8% without anything raising a flag.
The bundled demo script also expects transformers plus trust_remote_code=True. My first run died on ModuleNotFoundError: No module named 'transformers' — an "official tool" that does not exist on your machine, for three lines of code you could have written yourself.
Three defaults in the tokenizers library
Once the counting script exists, the mistakes stop coming from tokenizers and start coming from library defaults.
One: encode() adds special tokens by default. Llama-3.1 and Mistral gain 1 per call (a BOS); Qwen3 and DeepSeek gain 0. Estimate one batch with tiktoken (no special tokens) and another with tokenizers (special tokens on) and the two numbers were never comparable to begin with. A million short strings quietly becomes a million extra tokens on the invoice.
Two: decode and skip_special_tokens. My first round-trip check called decode(ids, skip_special_tokens=False) and reported 13 of 13 cases mismatched, English included. The tokenizer was fine. The BOS had been decoded literally as <|begin_of_text|> and prepended to the text. With the default it is 13 of 13 correct. Same rule I keep writing down: suspect the check before you suspect the thing being checked.
Three: the two libraries do not share an interface. tiktoken's encode() returns a list; tokenizers returns an Encoding object and you must take .ids. Hand a list to tokenizers and you get TypeError: TextInputSequence must be str. Build one adapter that runs every tokenizer and these differences have to be normalized there, or the failures show up somewhere much stranger.
Numbers, full-width characters, UUIDs
The expensive characters in technical load are not always Chinese. On structured input the vendors disagree more than they do on prose:
| Input | cl100k | o200k | Qwen3 | DeepSeek | Mistral |
|---|---|---|---|---|---|
1284500 | 3 | 3 | 7 | 3 | 8 |
1,284,500 | 5 | 5 | 9 | 5 | 10 |
| UUID (36 chars) | 23 | 23 | 33 | 23 | 34 |
full-width ABC123 | 9 | 5 | 6 | 5 | 19 |
half-width ABC123 | 2 | 2 | 4 | 2 | 4 |
| one markdown table row | 28 | 25 | 30 | 23 | 30 |
打包失败 😅 重跑三次 ✅ | 13 | 10 | 10 | 9 | 16 |
Qwen3 splits 1284500 into 7 pieces — 2.33 times cl100k — and charges 43.5% more for a UUID. DeepSeek matches cl100k exactly on digits and comes in cheaper on the table row (23 against 28).
This lands hard on teams doing log analysis, order-ID extraction or model-number matching. If half your payload is IDs and digits, a Qwen-family bill will run well above whatever a Chinese-character ratio predicted. Full-width characters are the other surprise: ABC123 is 5 tokens on o200k and 19 on Mistral, a 3.8x gap — and full-width letters typed by a Chinese IME land in prompts all the time.
Three rules for costing a project
Use the tokenizer of the model you are calling. Not the closest one. cl100k is the default I keep seeing in scripts, and on Chinese it overestimates DeepSeek by 73.0% and Qwen3 by 58.1%.
Sample real requests, not homogeneous text. My technical article corpus is 3,157 Chinese characters across 6,066 total characters; the six tokenizers return 5,531, 4,101, 4,278, 4,072, 3,613 and 5,738 — a 58.8% spread. Smaller than the 73.0% on pure Chinese, because the ASCII in a technical document flattens the difference. A pure-Chinese sample inflates the gap; a pure-code sample hides it.
Take both multipliers from their own vendor. This is where budgets actually break. Per the official pricing pages as of this writing: DeepSeek flash input is $0.30 per million tokens at peak and $0.15 off-peak; gpt-4o input is $2.50 per million tokens.
Per 100,000 Chinese characters: DeepSeek needs 62,712 tokens, which costs $0.0188. gpt-4o needs 108,475 tokens, which costs $0.2712. The real ratio is 14.4x. Count both vendors with cl100k and you get 8.3x — the price ratio is right, the token-count difference cancels out, and a real 14.4x gap gets recorded as 8.3x.
Troubleshooting: five errors I actually hit
Error one: the official demo script needs transformers
ModuleNotFoundError: No module named 'transformers'
The script shipped inside the vendor's tokenizer bundle imports transformers on line one, then calls AutoTokenizer.from_pretrained(..., trust_remote_code=True). If you do not want that dependency, load tokenizer.json directly with Tokenizer.from_file() and skip it entirely.
Error two: gated repository returns 401
Cannot access gated repo for url https://huggingface.co/meta-llama/Llama-3.1-8B/resolve/main/tokenizer.json.
Access to model meta-llama/Llama-3.1-8B is restricted. You must have access to it and be
authenticated to access it. Please log in.
This is a repository gate, not a network problem. An ungated mirror resolves it, but say so in your write-up — that is why the Llama row in my table is labelled as a mirror.
Error three: from_file pointed at a directory
Exception : 拒绝访问。 (os error 5)
Tokenizer.from_file() was handed a directory. The message never says "not a file"; it surfaces as an OS-level access denial, localized into Chinese on my machine, which makes it nearly impossible to grep in logs. Pass the full path to tokenizer.json.
Error four: 13 of 13 round-trip mismatches, all false
Not an error at all — my check was wrong. decode(ids, skip_special_tokens=False) decoded the BOS into a literal <|begin_of_text|> prefix, so joined != text for every case. It reads exactly like a tokenizer dropping characters. With the default argument it is 13 of 13 correct.
Error five: handing a list to encode
TypeError: TextInputSequence must be str
tokenizer.encode() takes a single string; batching goes through encode_batch(). tiktoken's encode() also refuses lists, so both sides need your own loop.
What I could not verify
I had no API key for this run, so I could not compare the local counts against the usage field a server returns. That distinction matters: every number here demonstrates relative differences between vocabularies, not the absolute number on your invoice.
To close that loop, send twenty real requests of increasing length and plot local counts against returned usage. If local runs systematically high, look for double-counted special tokens. If it runs low, look for the system prompt and chat template overhead you are not counting — that block differs per vendor and never appears in your own text.
Boundaries
I did not test Korean- or Japanese-specialized vocabularies, which move this line by another order of magnitude. I did not test multimodal input. I did not count chat-template overhead, only plain text. And vocabularies change with model versions: every number here is bound to tiktoken 0.14.0, tokenizers 0.23.3, CPython 3.13.14 and the tokenizer.json files I downloaded on 2026-11-02.
The same failure shape shows up elsewhere. I ran into it last week with serialization libraries (json, orjson and msgspec: nine defaults): different mechanism, identical nature — defaults are the most expensive implicit contract in a library. And how tokens reach the front end is yet another layer; the streaming piece (LangGraph streaming: three entry points) treats counting and transport as separate concerns for exactly this reason.
FAQ
Q: Can I just use tiktoken's o200k for every model?
No. On Chinese it costs 0.814 tokens per character — 29.8% above DeepSeek's 0.627 and 18.7% above Qwen3's 0.686. It beats cl100k, but it is nobody's stand-in.
Q: Does a bigger vocabulary always mean a smaller bill?
No. Qwen3 has the largest vocabulary here (151,669) and is not the cheapest on Chinese. What matters is how much Chinese the training corpus put inside that vocabulary, and vendors rarely publish it. Counting real text is the only reliable path.
Q: My local count is lower than the bill. Why?
Check special tokens and the chat template first. System prompts, role markers and tool definitions all consume tokens, and none of them are in the text you counted.
Q: Does the vocabulary change between versions of the same model?
Yes. DeepSeek's official bundle and the HF V3 directory matched exactly in my run, but that is luck, not a guarantee. Re-run the comparison after every model upgrade.
No affiliate links in this article. Every figure comes from local runs; the scripts and corpus definitions are described above, and official prices are as published on the vendors' pricing pages at the time of writing.
👉 Join Xiaomi MiMo Platform: Leading AI model platform with cost-effective inference
👉 Join Aliyun AI: Top AI products with exclusive coupons for business innovation
📌 This article was AI-assisted generated and human-reviewed | TechPassive — An AI-driven content testing site focused on real tool reviews
🔗 Recommended Tools
These are carefully selected tools. Using our affiliate links supports us to keep producing quality content:
As an Amazon Associate, I earn from qualifying purchases at no extra cost to you.