Why Identical-Looking Strings Aren't Equal: 7 Unicode Traps
TL;DR: This is not a mojibake problem. It is a "two strings that look identical to a human are stored as different bytes" problem. I actually ran seven probes on Windows 11 with CPython 3.13.14, and all seven traps share one property: nothing raises, the logs stay clean, and the exit code is 0 until a filename stops opening or a unique index stops blocking duplicates.
- NFC versus NFD: the same "é" can be one code point or two, 2 bytes versus 3 in UTF-8, and
==returns False - NFC is not closed under concatenation: two individually normalized fields, joined, may not be NFC anymore
- NFKC rewrites your data: "①" becomes "1", "½" becomes three characters, "℃" becomes "°C"
- Zero-width characters and BOMs:
strip()will not remove them, dict lookups miss,PRIMARY KEYwill not stop them - Case mapping is not one-to-one:
'ß'.upper()grows from 1 character to 2,'İ'.lower()expands to 2 code points sorted()is not pinyin order: five surnames come out in an order no Chinese reader expects- Lone surrogates:
jsonreads them happily, then writing UTF-8 raisesUnicodeEncodeError
If you only take one rule away: store NFC, compare with NFKC plus casefold, let the display layer be the only place that "looks the same", and never treat "looks the same" as "is equal".
First, what a string actually is
The Unicode HOWTO opens with "A Unicode string is a sequence of code points". A string is a sequence of code points, not the glyphs your eyes see. That single sentence explains three things people routinely get wrong.
First, len() counts code points. In 'café', if the é is written as e plus U+0301 (combining acute accent), the length is 5 instead of 4.
Second, one perceived character can be several code points. On my machine '👨👩👧👦' has len() 7, '🇨🇳' has 2, the Thai 'กำ' has 2, and 'e' plus U+0301 has 2.
Third, identical code points are the only route to identical bytes, and identical bytes are the only route to "finding the same thing" in a filesystem, a database, or an HTTP header. Break any link in that chain and you get no exception, only "it exists but I cannot find it".
"One text, many representations" also happens one level up. The same Chinese paragraph tokenizes into counts that differ by more than 70% depending on the vocabulary, and the invoice follows the difference, which is the same class of problem (see five tokenizers, one Chinese paragraph).
The Python documentation for unicodedata.normalize contains a sentence that is basically this article's thesis:
Even if two unicode strings are normalized and look the same to a human reader, if one has combining characters and the other doesn't, they may not compare equal.
Read that again: already normalized strings can still compare unequal, because there are four normal forms and both NFC and NFD count as normalized. They simply picked different representations.
Trap 1: one filename, two byte sequences
I created café.txt in a temp directory using NFC, then tried to read it back using the NFD spelling of the exact same name:
FileNotFoundError: [Errno 2] No such file or directory:
'C:\\Users\\...\\Temp\\uni1-k5ep8ynt\\cafe\u0301.txt'
At the same moment os.listdir() returned the NFC form, and os.path.exists() returned True for the NFC path and False for the NFD path.
The cause is mundane: filesystems match names byte by byte. NFC's "é" is U+00E9, one code point, two UTF-8 bytes (C3 A9). NFD's "é" is U+0065 U+0301, two code points, three UTF-8 bytes (65 CC 81). Neither NTFS nor ext4 normalizes, so these are two different names that can coexist.
This bites hardest when files move between systems. If any step in the middle normalizes (an export, a repack, a sync tool, an editor saving the tree), half the files appear to vanish, and the error only shows up at the moment someone opens one.
Fix: before any "find by name" operation, normalize both sides to the same form.
Trap 2: NFC is not closed under concatenation
This is the one people miss, because it violates intuition: if every field was normalized, surely the concatenation is normalized too?
No. UAX #15, section 1.4:
none of the Normalization Forms are closed under string concatenation. That is, even if two strings X and Y are normalized, their string concatenation X+Y is not guaranteed to be normalized.
One of the standard's own examples is Hangul: two individually normalized jamo (U+1100 leading consonant, U+1161 vowel) should compose into a single syllable (U+AC00) once joined. I reproduced it:
NFC(x) = '\u1100' NFC(y) = '\u1161'
joined = NFC(x) + NFC(y) -> length 2
unicodedata.normalize('NFC', joined) -> length 1
unicodedata.is_normalized('NFC', joined) -> False
Latin does the same thing: 'a' and U+0301 are each NFC on their own, but is_normalized('NFC', ...) returns False after concatenation, because the pair should have composed into 'á'.
UAX #15 notes that all normal forms are closed under substringing; only concatenation breaks the guarantee. So the very common pipeline "normalize each field, concatenate, store" fails at the concatenation step.
Fix: normalize the final value, after joining, formatting, and template rendering. One pass at the input boundary does not stay valid downstream. Version strings have the same shape of problem: 1.0 and 1.0.0 look like the same thing, but comparison depends on how the constraint was written (see PEP 440 pinning, one character too few).
Trap 3: NFKC rewrites your data
Everyone learns "use NFKC for full-width to half-width conversion", and then uses it as a general-purpose cleaner. That step changes the data itself.
I measured 11 common characters; NFKC altered both content and length:
| Input | Code point | NFKC result | Length |
|---|---|---|---|
| ① | U+2460 | 1 | 1 to 1 (meaning changed) |
| ½ | U+00BD | 1⁄2 | 1 to 3 (up 200%) |
| ℃ | U+2103 | °C | 1 to 2 |
| ㈱ | U+3231 | (株) | 1 to 3 |
| Ⅻ | U+216B | XII | 1 to 3 |
| ㎡ | U+33A1 | m2 | 1 to 2 |
| … | U+2026 | ... | 1 to 3 |
| ⑩ | U+2469 | 10 | 1 to 2 |
| ™ | U+2122 | TM | 1 to 2 |
| A | U+FF21 | A | 1 to 1 |
| カ | U+FF76 | カ | 1 to 1 |
Two details matter. First, ½ does not become ASCII 1/2; it becomes 1 plus U+2044 (fraction slash) plus 2, three characters. If you later pass that to int() or use it as a filename, behavior is completely different. Second, NFC leaves every one of these untouched (for each of the 11, normalize('NFC', s) == s was True in my probe). "I want compatibility folding" and "I want to preserve the original" are opposite directions, and picking the wrong one produces no error, only permanent information loss.
UAX #15 section 1.2 is explicit:
Normalization Forms KC and KD must not be blindly applied to arbitrary text. Because they erase many formatting distinctions, they will prevent round-trip conversion to and from many legacy character sets.
Fix: NFKC is for derived values only, comparison keys and search terms. Never write it back over the original field. Store originals as NFC.
Trap 4: zero-width characters and BOMs
U+200B ZERO WIDTH SPACE is the most common "invisible to humans, decisive to machines" character. It arrives by copy-paste from a web page, from rich text, or from an Excel export.
Measured: 'user\u200bname' has length 9, == 'username' is False, and .strip() leaves the length at 9, because strip() only removes whitespace and U+200B is not whitespace. What you get next is an error that points nowhere near the cause:
KeyError: 'user\u200bname'
BOMs are subtler. The UTF-8 BOM is EF BB BF. The codecs docs state: "On encoding the utf-8-sig codec will write 0xef, 0xbb, 0xbf as the first three bytes to the file", and the same page says using a BOM in UTF-8 is "discouraged and should generally be avoided". The problem is that utf-8-sig only skips a BOM at the start of the file. Concatenate ten-megabyte chunks, or join several CSVs end to end, and the second file's BOM lands in front of its header row and becomes data.
Measured: reading a BOM-prefixed CSV as utf-8 yields field names ['\ufeffid', 'name'], so row['id'] raises KeyError immediately; utf-8-sig yields ['id', 'name']. Put the same character in a numeric column and you get:
ValueError: invalid literal for int() with base 10: '\ufeff123'
One reversal is worth knowing: int('123') succeeds (123) and float('1.5') returns 1.5. Python numeric parsing accepts full-width digits but rejects a BOM. So "full-width digits pass validation" is itself a trap that lets pasted full-width numbers into your database.
Trap 5: case mapping is not one-to-one
lower() and upper() do not guarantee the length stays the same, and they do not guarantee a round trip. Measured on my machine:
'ß'.upper()gives'SS', length 1 to 2'fi'.upper()gives'FI', length 1 to 2'İ'.lower()gives'i'plus U+0307, length 1 to 2, and it is not equal to'i''ΣΣ'.lower()gives'σς': the same Σ becomes final sigma at the end and medial sigma at the start, so the result depends on position'ß'.casefold()gives'ss', and'ẞ'.casefold()also gives'ss'
The Python docs describe casefold() as "Casefolding is similar to lowercasing but more aggressive", with the example 'straße'.casefold() returning 'strasse'. In other words, case-insensitive comparison should use casefold, not lower, otherwise 'straße' and 'STRASSE' are judged different.
The nastier variant is length change plus indexing: call upper() first, then slice by position (say, the first three characters for a short code), and a single 'ß' shifts every subsequent index by one. UAX #15 section 10 states the general rule: "Higher-level processes that transform or compare strings, or that perform other higher-level functions, must respect canonical equivalence or problems will result."
Fix: compare with casefold(). Do every position-based operation on the original string, never on the converted length.
Trap 6: sorted() is not pinyin order
sorted() orders by code point, which for Chinese has nothing to do with pinyin or stroke count. Five surnames measured:
sorted(['陈','阿','白','张','李']) -> 张 李 白 阿 陈
locale.setlocale('Chinese_China.936') + strxfrm -> 阿 白 陈 李 张
Only the second line matches what a Chinese reader expects. That is also the point: sorting is a locale concern, not a string-library concern. locale.strxfrm hands the comparison key to the platform C library, so the same code needs Chinese_China.936 on Windows and usually zh_CN.UTF-8 on POSIX, and raises locale.Error when the name is unavailable.
On the database side, SQLite ships exactly three collations: BINARY, NOCASE, and RTRIM. The docs say "BINARY - Compares string data using memcmp()", and for NOCASE, "only ASCII characters are case folded". Handing Chinese sorting to a default ORDER BY therefore gives you code point order as well.
Fix: put sorting in the presentation layer or in a database collation, keep only equality in business logic, and if you must sort yourself, name the locale explicitly and degrade to code point order with a warning when it is unavailable. Time fields show the identical symptom: two instants that look the same still refuse to compare equal (see zoneinfo and the six silent DST errors).
Trap 7: lone surrogates, readable but unwritable
This is the one that makes people conclude "the data is corrupt". json.loads('"\\ud800"') returns a one-character string on CPython 3.13.14 whose repr is '\ud800'. That is a lone surrogate, not a real character. Then:
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800'
in position 0: surrogates not allowed
Writing a file fails the same way, because open(..., encoding='utf-8') calls encode internally. Meanwhile json.dumps({"k": "\ud800"}) happily emits {"k": "\ud800"}, so the JSON that "reads and writes fine" explodes somewhere downstream the moment it has to become UTF-8 bytes.
The usual source of such strings is the surrogateescape error handler. The Unicode HOWTO says it will "decode any non-ASCII bytes as code points in a special range running from U+DC80 to U+DCFF". When you read a legacy file with errors='surrogateescape', undecodable bytes become surrogates in that range. That makes read-modify-write lossless, but the moment you encode('utf-8') that string, it is rejected.
Fix: keep two categories apart. Internal binary proxies may only live inside a surrogateescape round trip. Before any boundary (JSON, HTTP, database, logs), verify with a plain s.encode('utf-8'): if it raises, the string is not fit to cross. Either accept the loss explicitly with errors='replace' and log it, or reject the record and keep the original bytes.
Which form where
| Layer | Use | Why |
|---|---|---|
| Storage (database, file content) | NFC | UAX #15 notes W3C and other specs recommend NFC for web content |
| Comparison key, unique index | NFKC plus casefold | Folds full-width and compatibility variants so the index can block duplicates |
| Display | Original value (NFC) | No compatibility folding, so "① becomes 1" cannot happen |
| Filename lookup | Same form on both sides (usually NFC) | Filesystems do not normalize, they compare bytes |
| Chinese sorting | Platform locale plus strxfrm | sorted() gives code point order |
| Pre-write validation | One encode('utf-8') attempt | The cheapest check that catches lone surrogates |
Three snippets you can copy
Comparison key:
import unicodedata
def compare_key(s: str) -> str:
# compatibility fold first, then case fold
return unicodedata.normalize("NFKC", s).casefold()
assert compare_key("ABC") == compare_key("abc")
assert compare_key("Straße") == compare_key("STRASSE")
Removing invisible characters:
import re
INVISIBLE = re.compile(r"[\u200b-\u200f\u202a-\u202e\u2060\ufeff]")
def strip_invisible(s: str) -> str:
return INVISIBLE.sub("", s)
assert strip_invisible("user\u200bname") == "username"
A batch audit to run before data lands. It only reports, it never mutates:
import unicodedata as ud
def audit(rows, columns):
bad = []
for i, row in enumerate(rows):
for col in columns:
v = row[col]
if not isinstance(v, str):
continue
if not ud.is_normalized("NFC", v):
bad.append((i, col, "not-nfc", repr(v)))
if strip_invisible(v) != v:
bad.append((i, col, "invisible", repr(v)))
try:
v.encode("utf-8")
except UnicodeEncodeError:
bad.append((i, col, "surrogate", repr(v)))
return bad
unicodedata.is_normalized has been available since 3.8, so there is no extra dependency. On a hundred thousand rows this normally finishes in under a second, which is far cheaper than debugging "why does this record never come back from a search".
To find these characters in bulk the cheapest tool is a regular expression, and regular expressions carry their own set of silent failures ($ producing two empty matches, \b behaving positionally inside CJK text, backtracking blowups), which you can read alongside eight regex matches that silently go wrong. There are no affiliate links in this post.
Troubleshooting: four real errors
Every error below was triggered on this machine while writing this piece, and the text is pasted verbatim rather than paraphrased into a "typical error".
Error 1: surrogates not allowed
Traceback (most recent call last):
File "probe.py", line 71, in module
open("x.txt", "w", encoding="utf-8").write(s)
UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800'
in position 0: surrogates not allowed
- Symptom: the field read from JSON looks fine, but writing it to a file or sending it over HTTP fails; the log shows the escape
\ud800, which looks harmless. - Cause:
json.loadsaccepts lone surrogates such as\uD800and Python'sstrallows them, but UTF-8 cannot encode surrogate code points. - Fix: validate at the boundary with
s.encode("utf-8"). If losing the character is acceptable, useerrors="replace"and log it. If the original bytes must survive, keep the whole pipeline onsurrogateescapefor I/O andsurrogatepassonly in memory, and never let it into JSON.
Error 2: a file that exists but cannot be found
FileNotFoundError: [Errno 2] No such file or directory:
'C:\\Users\\...\\Temp\\uni1-k5ep8ynt\\cafe\u0301.txt'
- Symptom:
os.listdir()clearly showscafé.txt, but building the path from a name fails; it appears after moving data between machines or after one sync run. - Cause: the directory entry is NFC, the path the code built is NFD, and both NTFS and ext4 compare bytes without normalizing.
- Fix: normalize both sides with
unicodedata.normalize("NFC", name)before comparing or looking up, including names that come from user input or configuration.
Error 3: a BOM inside a numeric column
ValueError: invalid literal for int() with base 10: '\ufeff123'
- Symptom: a CSV import says "not a number" while the column looks entirely numeric, and
row['id']raises KeyError as well. - Cause: a BOM ended up in the middle of the file through concatenation or a repeated export, and the
utf-8decoder does not skip it there, so it becomes the first character of a header or value. - Fix: read CSVs with
encoding="utf-8-sig"(it only skips a leading BOM), runstrip_invisible()before loading, and export without a BOM, which is what the documentation recommends for UTF-8 anyway.
Error 4: a legacy file that refuses UTF-8
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd6
in position 0: invalid continuation byte
- Symptom: most files in an export batch read fine, a few fail on the first line.
- Cause: those files are GBK, where the two-character word for Chinese is the byte pair
D6 D0 CE C4;0xD6is not a valid UTF-8 leading byte. - Fix: do not guess. Detect the encoding and declare it explicitly, or read with
errors="surrogateescape", edit only the ASCII parts, and write back unchanged. The test is whether the bytes survive a round trip: if they do, do not transcode.
FAQ
Q: If NFKC can turn full-width into half-width, why not use it to normalize usernames?
Use it, but only for the comparison key. Store the username as the user typed it (NFC) and fold both sides with NFKC plus casefold at comparison time. Overwriting the stored value turns ½ into three characters and ① into 1, which the user will see on their next login, and the change is irreversible.
Q: If I normalize every input to NFC, am I safe?
No, for two reasons. Concatenation is not closed, so field-level normalization can still produce a non-NFC value after joining (Trap 2). And NFC does not resolve compatibility variants: ① and 1, or full-width A and A, remain different characters under NFC. Normalization removes canonical differences only; whether to fold compatibility differences is a product decision you have to make explicitly.
Q: My database already contains dirty rows. How do I find them?
Run the audit snippet from the previous section and count results by not-nfc, invisible, and surrogate. Repair in two stages: first NFC and invisible-character cleanup, which is low risk, then evaluate the merge conflicts that NFKC folding would create. If the result feeds a unique index, count the rows that would collide before choosing a merge policy.
Boundaries of this write-up
Stating the limits is more useful than adding one more conclusion.
- Measured environment: Windows 11 with CPython 3.13.14,
unicodedata.unidata_version15.1.0, NTFS filesystem. Every number here comes from probes on this machine, not from someone else's write-up. - The claim that macOS filesystems store filenames in NFD is not something I could test here, so it is not stated as a conclusion. If you move files across macOS, apply the Trap 1 fix; you do not need to know how the other side stores them.
- The Unicode data version inside
unicodedatafollows the Python build: 15.1.0 here, while the current documentation page is based on a newer UCD. Normalization results are stable across versions, as UAX #15 section 3 puts it: "if a string that does not have any unassigned characters is normalized under one version of Unicode, it must remain normalized under all future versions of Unicode". What changes is which code points count as assigned, so logunicodedata.unidata_versioninstead of hard-coding a version in your checks. - All four normal forms are idempotent (UAX #15 section 7 lists it as a design goal, and
normalize(form, normalize(form, s))matched for all four forms in my probe). That only means repeated normalization is harmless; it does not mean one pass is enough forever, and Trap 2 is the counterexample.
If you are chasing a bug that smells like "the text field does not match", resist the urge to refactor first. Print both values with repr(), compare code points, and within five minutes you will land on one of the seven traps above. That is cheaper than reading a hundred lines of normalization code.
👉 Join Xiaomi MiMo Platform: Leading AI model platform with cost-effective inference
👉 Join Aliyun AI: Top AI products with exclusive coupons for business innovation
📌 This article was AI-assisted generated and human-reviewed | TechPassive — An AI-driven content testing site focused on real tool reviews
🔗 Recommended Tools
These are carefully selected tools. Using our affiliate links supports us to keep producing quality content:
As an Amazon Associate, I earn from qualifying purchases at no extra cost to you.