← Back to Home

Why Identical-Looking Strings Aren't Equal: 7 Unicode Traps

UnicodeNFCencoding

TL;DR: This is not a mojibake problem. It is a "two strings that look identical to a human are stored as different bytes" problem. I actually ran seven probes on Windows 11 with CPython 3.13.14, and all seven traps share one property: nothing raises, the logs stay clean, and the exit code is 0 until a filename stops opening or a unique index stops blocking duplicates.

If you only take one rule away: store NFC, compare with NFKC plus casefold, let the display layer be the only place that "looks the same", and never treat "looks the same" as "is equal".

First, what a string actually is

The Unicode HOWTO opens with "A Unicode string is a sequence of code points". A string is a sequence of code points, not the glyphs your eyes see. That single sentence explains three things people routinely get wrong.

First, len() counts code points. In 'café', if the é is written as e plus U+0301 (combining acute accent), the length is 5 instead of 4.

Second, one perceived character can be several code points. On my machine '👨‍👩‍👧‍👦' has len() 7, '🇨🇳' has 2, the Thai 'กำ' has 2, and 'e' plus U+0301 has 2.

Third, identical code points are the only route to identical bytes, and identical bytes are the only route to "finding the same thing" in a filesystem, a database, or an HTTP header. Break any link in that chain and you get no exception, only "it exists but I cannot find it".

"One text, many representations" also happens one level up. The same Chinese paragraph tokenizes into counts that differ by more than 70% depending on the vocabulary, and the invoice follows the difference, which is the same class of problem (see five tokenizers, one Chinese paragraph).

The Python documentation for unicodedata.normalize contains a sentence that is basically this article's thesis:

Even if two unicode strings are normalized and look the same to a human reader, if one has combining characters and the other doesn't, they may not compare equal.

Read that again: already normalized strings can still compare unequal, because there are four normal forms and both NFC and NFD count as normalized. They simply picked different representations.

Trap 1: one filename, two byte sequences

I created café.txt in a temp directory using NFC, then tried to read it back using the NFD spelling of the exact same name:

FileNotFoundError: [Errno 2] No such file or directory:
'C:\\Users\\...\\Temp\\uni1-k5ep8ynt\\cafe\u0301.txt'

At the same moment os.listdir() returned the NFC form, and os.path.exists() returned True for the NFC path and False for the NFD path.

The cause is mundane: filesystems match names byte by byte. NFC's "é" is U+00E9, one code point, two UTF-8 bytes (C3 A9). NFD's "é" is U+0065 U+0301, two code points, three UTF-8 bytes (65 CC 81). Neither NTFS nor ext4 normalizes, so these are two different names that can coexist.

This bites hardest when files move between systems. If any step in the middle normalizes (an export, a repack, a sync tool, an editor saving the tree), half the files appear to vanish, and the error only shows up at the moment someone opens one.

Fix: before any "find by name" operation, normalize both sides to the same form.

Trap 2: NFC is not closed under concatenation

This is the one people miss, because it violates intuition: if every field was normalized, surely the concatenation is normalized too?

No. UAX #15, section 1.4:

none of the Normalization Forms are closed under string concatenation. That is, even if two strings X and Y are normalized, their string concatenation X+Y is not guaranteed to be normalized.

One of the standard's own examples is Hangul: two individually normalized jamo (U+1100 leading consonant, U+1161 vowel) should compose into a single syllable (U+AC00) once joined. I reproduced it:

NFC(x) = '\u1100'   NFC(y) = '\u1161'
joined = NFC(x) + NFC(y)                    -> length 2
unicodedata.normalize('NFC', joined)        -> length 1
unicodedata.is_normalized('NFC', joined)    -> False

Latin does the same thing: 'a' and U+0301 are each NFC on their own, but is_normalized('NFC', ...) returns False after concatenation, because the pair should have composed into 'á'.

UAX #15 notes that all normal forms are closed under substringing; only concatenation breaks the guarantee. So the very common pipeline "normalize each field, concatenate, store" fails at the concatenation step.

Fix: normalize the final value, after joining, formatting, and template rendering. One pass at the input boundary does not stay valid downstream. Version strings have the same shape of problem: 1.0 and 1.0.0 look like the same thing, but comparison depends on how the constraint was written (see PEP 440 pinning, one character too few).

Trap 3: NFKC rewrites your data

Everyone learns "use NFKC for full-width to half-width conversion", and then uses it as a general-purpose cleaner. That step changes the data itself.

I measured 11 common characters; NFKC altered both content and length:

InputCode pointNFKC resultLength
①U+246011 to 1 (meaning changed)
½U+00BD1⁄21 to 3 (up 200%)
℃U+2103°C1 to 2
㈱U+3231(株)1 to 3
ⅫU+216BXII1 to 3
㎡U+33A1m21 to 2
…U+2026...1 to 3
⑩U+2469101 to 2
™U+2122TM1 to 2
AU+FF21A1 to 1
カU+FF76カ1 to 1

Two details matter. First, ½ does not become ASCII 1/2; it becomes 1 plus U+2044 (fraction slash) plus 2, three characters. If you later pass that to int() or use it as a filename, behavior is completely different. Second, NFC leaves every one of these untouched (for each of the 11, normalize('NFC', s) == s was True in my probe). "I want compatibility folding" and "I want to preserve the original" are opposite directions, and picking the wrong one produces no error, only permanent information loss.

UAX #15 section 1.2 is explicit:

Normalization Forms KC and KD must not be blindly applied to arbitrary text. Because they erase many formatting distinctions, they will prevent round-trip conversion to and from many legacy character sets.

Fix: NFKC is for derived values only, comparison keys and search terms. Never write it back over the original field. Store originals as NFC.

Trap 4: zero-width characters and BOMs

U+200B ZERO WIDTH SPACE is the most common "invisible to humans, decisive to machines" character. It arrives by copy-paste from a web page, from rich text, or from an Excel export.

Measured: 'user\u200bname' has length 9, == 'username' is False, and .strip() leaves the length at 9, because strip() only removes whitespace and U+200B is not whitespace. What you get next is an error that points nowhere near the cause:

KeyError: 'user\u200bname'

BOMs are subtler. The UTF-8 BOM is EF BB BF. The codecs docs state: "On encoding the utf-8-sig codec will write 0xef, 0xbb, 0xbf as the first three bytes to the file", and the same page says using a BOM in UTF-8 is "discouraged and should generally be avoided". The problem is that utf-8-sig only skips a BOM at the start of the file. Concatenate ten-megabyte chunks, or join several CSVs end to end, and the second file's BOM lands in front of its header row and becomes data.

Measured: reading a BOM-prefixed CSV as utf-8 yields field names ['\ufeffid', 'name'], so row['id'] raises KeyError immediately; utf-8-sig yields ['id', 'name']. Put the same character in a numeric column and you get:

ValueError: invalid literal for int() with base 10: '\ufeff123'

One reversal is worth knowing: int('123') succeeds (123) and float('1.5') returns 1.5. Python numeric parsing accepts full-width digits but rejects a BOM. So "full-width digits pass validation" is itself a trap that lets pasted full-width numbers into your database.

Trap 5: case mapping is not one-to-one

lower() and upper() do not guarantee the length stays the same, and they do not guarantee a round trip. Measured on my machine:

The Python docs describe casefold() as "Casefolding is similar to lowercasing but more aggressive", with the example 'straße'.casefold() returning 'strasse'. In other words, case-insensitive comparison should use casefold, not lower, otherwise 'straße' and 'STRASSE' are judged different.

The nastier variant is length change plus indexing: call upper() first, then slice by position (say, the first three characters for a short code), and a single 'ß' shifts every subsequent index by one. UAX #15 section 10 states the general rule: "Higher-level processes that transform or compare strings, or that perform other higher-level functions, must respect canonical equivalence or problems will result."

Fix: compare with casefold(). Do every position-based operation on the original string, never on the converted length.

Trap 6: sorted() is not pinyin order

sorted() orders by code point, which for Chinese has nothing to do with pinyin or stroke count. Five surnames measured:

sorted(['陈','阿','白','张','李'])                    -> 张 李 白 阿 陈
locale.setlocale('Chinese_China.936') + strxfrm       -> 阿 白 陈 李 张

Only the second line matches what a Chinese reader expects. That is also the point: sorting is a locale concern, not a string-library concern. locale.strxfrm hands the comparison key to the platform C library, so the same code needs Chinese_China.936 on Windows and usually zh_CN.UTF-8 on POSIX, and raises locale.Error when the name is unavailable.

On the database side, SQLite ships exactly three collations: BINARY, NOCASE, and RTRIM. The docs say "BINARY - Compares string data using memcmp()", and for NOCASE, "only ASCII characters are case folded". Handing Chinese sorting to a default ORDER BY therefore gives you code point order as well.

Fix: put sorting in the presentation layer or in a database collation, keep only equality in business logic, and if you must sort yourself, name the locale explicitly and degrade to code point order with a warning when it is unavailable. Time fields show the identical symptom: two instants that look the same still refuse to compare equal (see zoneinfo and the six silent DST errors).

Trap 7: lone surrogates, readable but unwritable

This is the one that makes people conclude "the data is corrupt". json.loads('"\\ud800"') returns a one-character string on CPython 3.13.14 whose repr is '\ud800'. That is a lone surrogate, not a real character. Then:

UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800'
in position 0: surrogates not allowed

Writing a file fails the same way, because open(..., encoding='utf-8') calls encode internally. Meanwhile json.dumps({"k": "\ud800"}) happily emits {"k": "\ud800"}, so the JSON that "reads and writes fine" explodes somewhere downstream the moment it has to become UTF-8 bytes.

The usual source of such strings is the surrogateescape error handler. The Unicode HOWTO says it will "decode any non-ASCII bytes as code points in a special range running from U+DC80 to U+DCFF". When you read a legacy file with errors='surrogateescape', undecodable bytes become surrogates in that range. That makes read-modify-write lossless, but the moment you encode('utf-8') that string, it is rejected.

Fix: keep two categories apart. Internal binary proxies may only live inside a surrogateescape round trip. Before any boundary (JSON, HTTP, database, logs), verify with a plain s.encode('utf-8'): if it raises, the string is not fit to cross. Either accept the loss explicitly with errors='replace' and log it, or reject the record and keep the original bytes.

Which form where

LayerUseWhy
Storage (database, file content)NFCUAX #15 notes W3C and other specs recommend NFC for web content
Comparison key, unique indexNFKC plus casefoldFolds full-width and compatibility variants so the index can block duplicates
DisplayOriginal value (NFC)No compatibility folding, so "① becomes 1" cannot happen
Filename lookupSame form on both sides (usually NFC)Filesystems do not normalize, they compare bytes
Chinese sortingPlatform locale plus strxfrmsorted() gives code point order
Pre-write validationOne encode('utf-8') attemptThe cheapest check that catches lone surrogates

Three snippets you can copy

Comparison key:

import unicodedata

def compare_key(s: str) -> str:
    # compatibility fold first, then case fold
    return unicodedata.normalize("NFKC", s).casefold()

assert compare_key("ABC") == compare_key("abc")
assert compare_key("Straße") == compare_key("STRASSE")

Removing invisible characters:

import re

INVISIBLE = re.compile(r"[\u200b-\u200f\u202a-\u202e\u2060\ufeff]")

def strip_invisible(s: str) -> str:
    return INVISIBLE.sub("", s)

assert strip_invisible("user\u200bname") == "username"

A batch audit to run before data lands. It only reports, it never mutates:

import unicodedata as ud

def audit(rows, columns):
    bad = []
    for i, row in enumerate(rows):
        for col in columns:
            v = row[col]
            if not isinstance(v, str):
                continue
            if not ud.is_normalized("NFC", v):
                bad.append((i, col, "not-nfc", repr(v)))
            if strip_invisible(v) != v:
                bad.append((i, col, "invisible", repr(v)))
            try:
                v.encode("utf-8")
            except UnicodeEncodeError:
                bad.append((i, col, "surrogate", repr(v)))
    return bad

unicodedata.is_normalized has been available since 3.8, so there is no extra dependency. On a hundred thousand rows this normally finishes in under a second, which is far cheaper than debugging "why does this record never come back from a search".

To find these characters in bulk the cheapest tool is a regular expression, and regular expressions carry their own set of silent failures ($ producing two empty matches, \b behaving positionally inside CJK text, backtracking blowups), which you can read alongside eight regex matches that silently go wrong. There are no affiliate links in this post.

Troubleshooting: four real errors

Every error below was triggered on this machine while writing this piece, and the text is pasted verbatim rather than paraphrased into a "typical error".

Error 1: surrogates not allowed

Traceback (most recent call last):
  File "probe.py", line 71, in module
    open("x.txt", "w", encoding="utf-8").write(s)
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800'
in position 0: surrogates not allowed

Error 2: a file that exists but cannot be found

FileNotFoundError: [Errno 2] No such file or directory:
'C:\\Users\\...\\Temp\\uni1-k5ep8ynt\\cafe\u0301.txt'

Error 3: a BOM inside a numeric column

ValueError: invalid literal for int() with base 10: '\ufeff123'

Error 4: a legacy file that refuses UTF-8

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd6
in position 0: invalid continuation byte

FAQ

Q: If NFKC can turn full-width into half-width, why not use it to normalize usernames?

Use it, but only for the comparison key. Store the username as the user typed it (NFC) and fold both sides with NFKC plus casefold at comparison time. Overwriting the stored value turns ½ into three characters and ① into 1, which the user will see on their next login, and the change is irreversible.

Q: If I normalize every input to NFC, am I safe?

No, for two reasons. Concatenation is not closed, so field-level normalization can still produce a non-NFC value after joining (Trap 2). And NFC does not resolve compatibility variants: ① and 1, or full-width A and A, remain different characters under NFC. Normalization removes canonical differences only; whether to fold compatibility differences is a product decision you have to make explicitly.

Q: My database already contains dirty rows. How do I find them?

Run the audit snippet from the previous section and count results by not-nfc, invisible, and surrogate. Repair in two stages: first NFC and invisible-character cleanup, which is low risk, then evaluate the merge conflicts that NFKC folding would create. If the result feeds a unique index, count the rows that would collide before choosing a merge policy.

Boundaries of this write-up

Stating the limits is more useful than adding one more conclusion.

If you are chasing a bug that smells like "the text field does not match", resist the urge to refactor first. Print both values with repr(), compare code points, and within five minutes you will land on one of the seven traps above. That is cheaper than reading a hundred lines of normalization code.

👉 Join Xiaomi MiMo Platform: Leading AI model platform with cost-effective inference

👉 Join Aliyun AI: Top AI products with exclusive coupons for business innovation

📌 This article was AI-assisted generated and human-reviewed | TechPassive — An AI-driven content testing site focused on real tool reviews

🔗 Recommended Tools

These are carefully selected tools. Using our affiliate links supports us to keep producing quality content:

As an Amazon Associate, I earn from qualifying purchases at no extra cost to you.

☁️ DigitalOcean Cloud ⚡ Vultr VPS 🤖 QoderWork CN (Refer & Earn) ☁️ Aliyun AI Products 📚 WordPress Books 🧠 AI & LLM Books 🏗️ System Design Books 🐳 Docker & K8s Books 🐧 Linux Admin Books 🐍 Python Books 🗄️ Database Books 🚀 DevOps & SRE Books 🤖 Xiaomi MiMo Platform
← Back to Home