← Back to Home

Five decode error handlers and the one that round-trips

Pythonencodingdebugging

A 47-byte file went through five scripts on my machine. Not one of the five wrote the same bytes back — except the one that used surrogateescape.

This is not an edge case. Every program that reads logs, CSV exports or vendor files eventually receives bytes that are not valid in the encoding it assumed: a table exported from a GBK system, a truncated multi-byte UTF-8 character, a stray byte at the end of a scanner dump. Python ships five standard error handlers, and the default is strict. So the usual fix is to switch it to ignore and let the program finish.

The problem is that "the program finished" and "the data survived" are two different things. ignore makes the run complete while deleting that byte from the result. replace turns it into U+FFFD. backslashreplace turns it into four printable characters. Only surrogateescape carries it through unchanged, and the price is that the resulting string can no longer be written back by ordinary means.

// FLOW_01Four gates a bad byte passes through4 NODES01STEP_01bytes arrive: file, pipe, socket, child process output02STEP_02pick an encoding: utf-8, or the local code page03STEP_03pick errors: one of five handlers, strict by default04STEP_04decide whether to write back: is this round trip byte-identicalEND_OF_FLOW
Four gates a bad byte passes through · diagram drawn from this article
Python
Technologies referenced · icons from Simple Icons (CC0)

errors= only decides what happens at the moment of failure

The official codecs documentation is explicit: errors specifies what to do when the conversion cannot follow the encoding's rules, and the default 'strict' raises a subclass of UnicodeError. It is not an encoding selector, and it does not check whether your encoding choice is right. That last part is what bites later.

Direction matters too. bytes.decode() and str.encode() have separate handler sets:

Use them the other way around and you get a raw TypeError. When I passed namereplace to a decode call, the message was TypeError: don't know how to handle UnicodeDecodeError in error callback — not a friendly "unsupported argument" hint.

One bad byte, five outputs

The sample matches what I see in real incidents: b'caf\xe9 ok', seven bytes, where the fourth is Latin-1 é and therefore invalid UTF-8.

bad = b'caf\xe9 ok'
for h in ('strict', 'ignore', 'replace', 'backslashreplace', 'surrogateescape'):
    print(h, repr(bad.decode('utf-8', h)))

The five results come back with lengths 6, 7, 10 and 7:

HandlerResultCharactersWhere the byte went
strictraises UnicodeDecodeError—never entered
ignore'caf ok'6deleted
replace'caf\ufffd ok'7became the replacement char
backslashreplace'caf\\xe9 ok'10became four characters
surrogateescape'caf\udce9 ok'7became U+DC E9

Note that replace and surrogateescape both return 7 characters. At a glance neither looks like it lost anything. That is the first place people misjudge this: matching lengths do not mean matching bytes.

Round trip: only one combination survives

Decoding is half the job; the write-back uses a handler of its own. Four decode handlers times three encode handlers gives twelve combinations, and exactly one is identical to the input:

Decodeencode strictencode surrogateescapeencode backslashreplace
ignore1 byte short1 byte short1 byte short
replace2 bytes longer2 bytes longer2 bytes longer
backslashreplace3 bytes longer3 bytes longer3 bytes longer
surrogateescaperaises UnicodeEncodeErroridentical5 bytes longer

The reason is simple. U+DC80 through U+DCFF is the range PEP 383 reserves for exactly this purpose. The documentation says bytes are replaced with surrogate code points in that range during decoding, and that the same handler turns them back into the same bytes during encoding. Every other handler destroys the original byte information at decode time, and no amount of clever encoding can recover it afterwards.

Writing back: who is changing your file size

I took the 47-byte sample with Chinese text plus two bad bytes, decoded it, and wrote it out again:

HandlerSize after writeIdentical to original
ignore45 Bno
replace51 Bno
backslashreplace53 Bno
surrogateescape47 Byes

The file grows under replace because U+FFFD encodes to three bytes in UTF-8, so each replaced byte adds two. When a log file creeps upward every night, an errors='replace' somewhere in the read path is a common root cause.

surrogateescape is not a repair, it is a delay

A string decoded with surrogateescape contains surrogate code points, which are not valid UTF-8. The documentation makes the scope clear: the filesystem encoding layer is the one that defaults to surrogateescape, ordinary text encodings do not.

So the same string may print fine (terminals tolerate it) and then fail the moment you write it:

s = b'caf\xe9 ok'.decode('utf-8', 'surrogateescape')
s.encode('utf-8')                      # UnicodeEncodeError
s.encode('utf-8', 'surrogateescape')   # fine, back to b'caf\xe9 ok'

CSV hides this well. I wrote two rows with csv.writer; writerows raised nothing and the text looked right. The failure arrived at encode:

UnicodeEncodeError: 'utf-8' codec can't encode character '\udce9' in position 14: surrogates not allowed

The correct pattern is symmetric: open(..., errors='surrogateescape') to read, the same handler to write. Using it on one side only moves the exception somewhere harder to diagnose. It is the same lesson I took from mapping CSV record boundaries: newline='' and errors are both paired switches, and setting one side only postpones the problem.

backslashreplace looks safest and pollutes twice

Its output is pure ASCII: 'caf\\xe9 ok'. That is seductive. str.isascii() returns True, the string survives JSON, logs and database writes without another encoding error, and every validator goes green.

The cost lands downstream. In 'price=\\xe9' the \xe9 is four visible characters, so any int() or float() call hits ValueError: invalid literal for int() with base 10: '\\xe9'. Worse, one byte has become four bytes, and those four characters encode to four bytes of their own, so a single conversion inflates the payload to four times its size, irreversibly.

JSON: the file is valid and the field is shorter

I ran the same b'{"id": 7, "name": "caf\xe9", "amount": 19.99}' through json.loads under all five handlers:

Handlername fieldJSON valid
strictraises UnicodeDecodeError—
ignore'caf'yes
replace'caf\ufffd'yes
backslashreplaceraises JSONDecodeErrorno
surrogateescape'caf\udce9'yes

Two counter-intuitive results. First, backslashreplace is the only handler that breaks JSON parsing, because \x is not a legal JSON escape; the message is Invalid \escape: line 1 column 23 (char 22). Second, ignore produces perfectly valid JSON that passes schema validation, with the name missing its last character and the amount untouched. The downstream system sees a normal customer name.

subprocess with text=True: the exception goes to a thread and stdout becomes None

This one convinces people the program is fine. A child process writes a bad byte to stdout, and the parent collects it with text=True:

r = subprocess.run([sys.executable, '-c', "import sys;sys.stdout.buffer.write(b'caf\xe9 ok')"],
                   capture_output=True, text=True)
print(r.returncode, r.stdout)

The output is 0 None. Return code zero, stdout is None rather than an empty string, and the main path continues. The real exception is stranded in the reader thread:

Exception in thread Thread-3 (_readerthread):
  File "subprocess.py", line 1615, in _readerthread
    buffer.append(fh.read())
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3

Passing errors='strict' explicitly changes nothing; only errors='surrogateescape' returns 'caf\udce9 ok'. text=True defaults to strict and never escalates a decode failure into CalledProcessError. It is the same pattern I documented in seven silent subprocess failures: at the child-process layer, failures default to quietly handing you less than you asked for.

On Windows the filesystem handler is not surrogateescape

The docs say an undecodable filesystem byte is handled with surrogateescape. That needs a caveat on Windows: on my machine sys.getfilesystemencodeerrors() returns 'surrogatepass', not 'surrogateescape'.

os.fsdecode(b'\xff')     # UnicodeDecodeError: invalid start byte
os.fsencode('\udcff')    # b'\xed\xb3\xbf', not b'\xff'

The PEP 383 round trip does not hold there. surrogatepass encodes the surrogate code point in its three-byte UTF-8 form instead of restoring the original single byte. If your code leans on os.fsdecode to absorb undecodable filenames, it will pass on Linux and raise on Windows. I covered the neighbouring layer in eight Windows path divergences; encoding sits directly downstream of it.

Three ways a custom handler dies

codecs.register_error expects a callable that returns a replacement and a new position. The documentation does not say the new position has to move forward. I wrote a handler straight from the docs that returned the old position, and the result was an infinite loop: five seconds of timeout, no exception, no output, the process still spinning. That is not a documentation gap, it simply is not checked. Advance the position by hand.

The other two failures are type errors, both visible only at runtime:

A handler that serves both directions needs isinstance(err.object, bytes) as its first line.

strict only guarantees a raise, not a correct decode

This is the part worth remembering. strict means "raise when the codec's rules are violated", and many codecs accept almost anything. I took ten common Chinese words, encoded them as UTF-8, and decoded the bytes as cp936 (the value of locale.getencoding() on this machine) with strict:

That is a 50% silent mistranslation rate, with no exception, no warning and roughly the same length. Deciding whether a decode is correct cannot rely on the absence of exceptions; it needs a round-trip comparison or a sampled human check. I ran into the same class of problem while mapping seven Unicode normalization traps.

A decision table by scenario

ScenarioUseWhy
Read someone else's file and write it backsurrogateescape on both sidesthe only byte-exact round trip
Read logs meant for humansreplacekeeps a visible marker, stable length
Feed JSON, XML or CSV parserssurrogateescape to locate, then fix the datareplace keeps the file valid and the field short
Encode for an old systemxmlcharrefreplace or namereplaceexplicit and readable, nothing vanishes
Binaries, images, archivesdo not decode at allerrors cannot fix "this was never text"

That last row is the precondition. errors handles "this is text but the bytes are invalid". It cannot handle "this is not text". Reading a PNG as UTF-8 fails no matter which handler you pick.

Two real errors and the order to debug them

Error one: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3: invalid continuation byte

The position is an offset inside the decode buffer, not a line number. I tested a four-line file where line three was broken, and the report said position 27 — the byte offset within the whole file, because the buffered reader decoded one whole block (all 38 bytes here) regardless of which line I was asking for. The debugging order: read the file with errors='surrogateescape', then re-encode line by line with the same handler and find the first line that raises. That gives you the line number directly.

Error two: UnicodeEncodeError: 'utf-8' codec can't encode character '\udce9' ... surrogates not allowed

A character in the U+DC80 to U+DCFF range is strong evidence that something upstream used errors='surrogateescape' (or that the value came from os.fsdecode). The fix is not to paper over it with errors='ignore'; go back upstream and check whether that handler was used symmetrically.

FAQ: four boundaries people ask about

Q: Can I just set errors='surrogateescape' globally?

A: On the read path, yes. On the write path it has to be paired, and any ordinary encode in between — json.dumps followed by .encode('utf-8'), a database driver, an HTTP request — will raise. It suits byte-preserving transport, not computation.

Q: Does replace break the JSON structure?

A: No. U+FFFD is a legal character and json.loads accepts it; what breaks is the field content. The handler that actually breaks JSON parsing is backslashreplace, because \x is not a legal escape.

Q: Why does the length barely change after ignore?

A: Dropping one three-byte character removes three bytes, which is invisible in a large file. But if an entire field is Chinese, ignore can turn it into an empty string, and the total size still looks fine.

Q: The docs say the filesystem uses surrogateescape, so why does Windows raise?

A: sys.getfilesystemencodeerrors() returns 'surrogatepass' here. Do not trust a remembered conclusion; print both sys.getfilesystemencoding() and sys.getfilesystemencodeerrors().

Closing

Of the five handlers, exactly one completes the round trip, and it demands that both ends of the read-write pair use the same value. The other four all have legitimate uses, but every one of them assumes you know what you are discarding. Before changing errors from strict to ignore, ask who downstream will assume the byte was never there.

This article contains no affiliate links. All numbers come from measurements on CPython 3.13.14 and Windows 11 (10.0.26300); the encoding semantics follow the codecs and Unicode HOWTO pages on docs.python.org.

👉 Join Xiaomi MiMo Platform: Leading AI model platform with cost-effective inference

👉 Join Aliyun AI: Top AI products with exclusive coupons for business innovation

📌 This article was AI-assisted generated and human-reviewed | TechPassive — An AI-driven content testing site focused on real tool reviews

🔗 Recommended Tools

These are carefully selected tools. Using our affiliate links supports us to keep producing quality content:

As an Amazon Associate, I earn from qualifying purchases at no extra cost to you.

☁️ DigitalOcean Cloud ⚡ Vultr VPS 🤖 QoderWork CN (Refer & Earn) ☁️ Aliyun AI Products 📚 WordPress Books 🧠 AI & LLM Books 🏗️ System Design Books 🐳 Docker & K8s Books 🐧 Linux Admin Books 🐍 Python Books 🗄️ Database Books 🚀 DevOps & SRE Books 🤖 Xiaomi MiMo Platform
← Back to Home