← Back to Home

Tokenizer Test: How Many Tokens Is One Chinese Character

tokenizertiktokenLLM

TL;DR: The widely quoted rule "1 token is approximately 4 characters" is an English estimate. Measured against 23 Chinese technical posts, it undercounts tokens by 3.4x. The deeper problem is byte-level BPE: a single Chinese character can be split into three byte fragments, and decoding tokens one by one silently destroys text (47 characters became 55, with 16 replacement characters) without raising a single exception.

I tested this with tiktoken 0.14.0 on this site's own corpus: 23 Chinese posts and 40 English posts, every number below reproduced on my machine. No affiliate links in this article.

The official wording is clear, nobody reads past the first line

The OpenAI help-center article on tokens stacks three sentences in a row. Verbatim:

That third sentence is the whole answer. The 4-character figure is an English-only approximation, and the same page says other languages behave differently. What circulates in engineering channels is only the first sentence, so a Chinese-language project starts its budget estimate wrong on day one.

My measurements: English technical prose came out at a median of 4.13 characters per token, essentially matching the official rule. Chinese technical prose came out at 0.841 tokens per character, meaning one token covers 1.19 Chinese characters. Applying 4 characters per token to Chinese computes 30% of the real token count, an undercount of 3.4x.

Five encodings, one word list twice the size

The public encodings available in tiktoken, with vocab sizes measured locally:

encodingn_vocabspecial tokenstypical models
gpt2 / r50k_base50,2571early GPT-2 lineage
p50k_base50,2811text-davinci-002/003
cl100k_base100,2775gpt-4, gpt-3.5-turbo, text-embedding-3
o200k_base200,0192gpt-4o, gpt-4.1, gpt-5, o1/o3/o4
o200k_harmony201,0881,091gpt-oss

The last row deserves a second look. o200k_harmony carries 1,091 special tokens, and the names give away its purpose: <|channel|>, <|message|>, <|constrain|>, <|call|>. If your counting script takes a permissive stance on allowed_special, this encoding will happily count a pile of control symbols as ordinary text.

What a bigger word list actually buys: 22.8% on Chinese, nothing on English

Re-running the same corpus under both encodings produced a result worth remembering:

Doubling the vocabulary buys almost entirely better merging for non-Latin scripts. English prose compression does not move at all. On a single 47-character Chinese sentence, cl100k_base produced 49 tokens and o200k_base produced 32.

That has a direct selection consequence. If your workload is mostly English, moving to an o200k model will not get cheaper because the vocabulary is bigger. If your workload is Chinese, the encoding switch alone saves roughly a fifth.

Three consequences of byte-level BPE

The atomic unit in BPE is the byte, not the character. One Chinese character occupies three bytes in UTF-8, and the vocabulary may not contain a whole-word entry for that character. When it does not, the character is split into two or three byte fragments, each becoming its own token.

Consequence one, counts exceed intuition: a 47-character Chinese passage can cost 49 tokens.

Consequence two, decoding tokens individually permanently loses text. This is the single most useful finding here. Decode each token of a Chinese sentence separately, then join the pieces:

toks = enc.encode_ordinary(text)          # 47 Chinese chars -> 49 tokens
joined = "".join(enc.decode([t]) for t in toks)
# 47 input characters -> 55 output characters, 16 U+FFFD, joined != text

Sixteen replacement characters mean sixteen Chinese characters were overwritten with �, and no exception was raised. To make it complain you have to ask explicitly:

enc.decode([toks[1]], errors="strict")
# UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 0-1: unexpected end of data

Consequence three, prefix decoding is not safe either. Decoding the running prefix toks[:i] eventually produces the correct final string, but 2 of 32 steps still flash a replacement character, because that step's last token lands mid-character. The robust approach is to hold back the incomplete trailing bytes and emit them once the next token completes the character. This is the same problem as streaming output; I hit it while measuring LangGraph's three streaming entry points, where token-level streaming to a browser always shows boxes for Chinese. The framework is not at fault, the decode boundary is.

Identifiers, timestamps and indentation are token sinks

The fields that blow up a prompt are rarely the prose:

samplecharacterstokenstokens/char
UUID 3f2b9c1e-7a4d-4f88-9b21-0c5e6d7a8b9036330.92
a JSON order object83400.48
date 2026-11-011060.60
time 06:10:46850.63

A UUID is the extreme case: 36 characters eat 33 tokens, nearly one to one. Prompts dense with log lines, trace IDs and order numbers will miss badly if you estimate from character count.

Numbers carry a counterintuitive twist too. 1284500 without separators costs 3 tokens; written 1,284,500 it costs 5, because each comma takes one. Formatting added for human readability is a net expense on the token bill.

Indentation behaves the same way. The same Python snippet costs 18 tokens with four-space indentation and 16 with tabs, an 11.1% gap that compounds across a large code corpus.

Special tokens raise by default

Feed <|endoftext|> in as ordinary text and tiktoken refuses:

ValueError: Encountered text corresponding to disallowed special token '<|endoftext|>'.
If you want this text to be encoded as a special token, pass it to `allowed_special`
To disable this check for all special tokens, pass `disallowed_special=()`.

The three escape hatches behave very differently. I ran each:

callresult
defaultraises ValueError
disallowed_special=()9 tokens (treated as text)
allowed_special="all"4 tokens (marker collapses to one control token)
encode_ordinary()9 tokens (same as the second row)

The five-token discrepancy is the visible part. The real problem is semantics: any user input containing a literal <|...|> sequence becomes an injected control token. Counting and encoding need two different policies, so do not collapse them into one helper.

Model names map to encodings by guessing

encoding_for_model is not a plain lookup, it also does prefix matching, and the source comment admits the weakness:

Note that this can match on non-existent models (e.g., gpt-3.5-turbo-FAKE)

Measured locally, gpt-3.5-turbo-FAKE returns cl100k_base with no error. Note also that the gpt-oss- prefix maps to o200k_harmony, and OpenAI's model page currently lists gpt-4o as Deprecated. A project that hardcodes gpt-4o will keep running fine and quietly produce wrong counts once that model is retired. When no mapping exists it is at least honest:

KeyError: 'Could not automatically map my-finetune-model to a tokeniser.
Please use `tiktoken.get_encoding` to explicitly get the tokeniser you expect.'

Converting to context window and dollars

OpenAI's model page for gpt-4o states "128,000 context window" and "16,384 max output tokens". Applying the measured ratios, 128,000 tokens holds roughly:

Writing the same idea in both languages (297 English characters / 56 tokens, versus 105 Chinese characters / 105 tokens under cl100k) gives a Chinese-to-English token ratio of 1.88x, and 1.34x under o200k. The failure mode is the same one I ran into while checking Prometheus quantile semantics: when the measurement convention is wrong, every downstream conclusion inherits the error.

At official standard-tier prices (per million tokens, as of publication): gpt-4o $2.50, gpt-4.1 $2.00, gpt-5 $1.25, gpt-4o-mini $0.15. One million Chinese characters then costs:

encodingtokens for 1M Chinese charsgpt-4o input cost
cl100k_base1,043,000$2.61
o200k_base681,000$1.70
4-chars-per-token guess250,000$0.63 (4.17x too low)

The last row is the textbook case of an estimator handing you a comfortable number. The gap between $0.63 and $2.61 will not be discovered by anyone before the invoice arrives. It belongs to the same family as the five assignments that silently stopped working in pandas 3.1 copy-on-write: a default that looks reasonable while the semantics changed underneath.

Troubleshooting: five real errors

Error one, special tokens rejected by default. ValueError: Encountered text corresponding to disallowed special token '<|endoftext|>'. This is a guard, not a bug. Fix it by separating concerns: encode_ordinary() for counting, an explicit allowlist for real encoding.

Error two, model name mapping fails. KeyError: 'Could not automatically map ... to a tokeniser.' Fix it by dropping the automatic lookup. Call tiktoken.get_encoding("o200k_base") directly and keep the model-to-encoding table in configuration, so a model upgrade forces a human decision.

Error three, strict decoding exposes a byte split. UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 0-1: unexpected end of data. Do not "fix" this by changing errors=; the message is telling you that this single token is not valid UTF-8 on its own.

Error four, and the most dangerous because it is silent: per-token decoding loses characters. 47 characters in, 55 out, 16 replacement characters, exit code 0. The only valid assertion is whether the rejoined string equals the input. Never rely on exceptions here.

Error five, runaway special-token counts. Switching to o200k_harmony moves the special-token count from 2 to 1,091. A counting script that lets special tokens through will report numbers that are mysteriously too small.

Scope of this article

Every token count here is a locally computed encoding value, not the usage reported by an API. The API adds message framing on top (role markers, delimiters); I have no API key, so that part is not measured and appears nowhere as a number. Also out of scope: Claude and Gemini vocabularies, the o200k_harmony chat template, and token accounting for images and audio. Prices come from the official pricing page, standard tier, as of November 1, 2026, and they change.

Five rules you can copy

1. Budget Chinese at 0.65 to 0.85 tokens per character. Keep 4.1 characters per token for English prose, where it holds.

2. Use encode_ordinary() on any counting path, and never enable allowed_special="all" there.

3. Buffer streaming output on UTF-8 boundaries instead of decoding tokens one at a time.

4. Count UUID-, timestamp- and JSON-heavy prompts separately; they cost more than twice what prose costs per character.

5. Write the model-to-encoding mapping as explicit configuration. Do not trust prefix matching.

👉 Join Xiaomi MiMo Platform: Leading AI model platform with cost-effective inference

👉 Join Aliyun AI: Top AI products with exclusive coupons for business innovation

📌 This article was AI-assisted generated and human-reviewed | TechPassive — An AI-driven content testing site focused on real tool reviews

🔗 Recommended Tools

These are carefully selected tools. Using our affiliate links supports us to keep producing quality content:

As an Amazon Associate, I earn from qualifying purchases at no extra cost to you.

☁️ DigitalOcean Cloud ⚡ Vultr VPS 🤖 QoderWork CN (Refer & Earn) ☁️ Aliyun AI Products 📚 WordPress Books 🧠 AI & LLM Books 🏗️ System Design Books 🐳 Docker & K8s Books 🐧 Linux Admin Books 🐍 Python Books 🗄️ Database Books 🚀 DevOps & SRE Books 🤖 Xiaomi MiMo Platform
← Back to Home