← Back to Home

Python file locking: the lock that holds nothing

Pythonfile lockingconcurrency

Two scheduled jobs wrote the same state file. By the end of the day the counter was 101 short. No traceback, no alert, and both processes exited with code 0, each one logging that the write had succeeded.

My first guess was filesystem caching. My second was to wrap the read-modify-write in a threading.Lock, which moved the number from 299 to 300: one hundred writes still gone. The reason is plain once you say it out loud. threading.Lock is an object inside one process, and the thing I needed to block was another process. It belongs to the same family as importing the wrong module or rounding money two different ways: the code ran, nothing raised, and the result is not what you asked for.

I ran all five experiments below on this machine: Windows 11 (10.0.26300), CPython 3.13.14 (MSC v.1944, 64-bit), measured on 2026-11-16. Standard library only, nothing installed. Every block is pasted verbatim. There are no affiliate links in this article and no commission links of any kind.

// FLOW_01The order I debug "two processes fighting over one file"6 NODES01STEP_01Numbers disagree but nothing raised02STEP_02check for cross-process exclusion at all03STEP_03check where the lock actually starts04STEP_04check whether the locked ranges overlap05STEP_05check that unlock matches the locked length06STEP_06check whether atomic replace is blocked by an open handleEND_OF_FLOW
The order I debug "two processes fighting over one file" · diagram drawn from this article
Two byte-range locks on one 64-byte file: process A locks bytes 0..7, process B locks 64..71 because its pointer sits at end of file
Two byte-range locks on one 64-byte file: process A locks bytes 0..7, process B locks 64..71 because its pointer sits at end of file · Concept illustration based on this article
PythonLinux
Technologies referenced · icons from Simple Icons (CC0)

The short version: only one of these three locks blocks a second process

threading.Lock blocks threads. fcntl.flock blocks processes on Linux but is only advisory. On Windows the thing that blocks a process is a byte-range lock from msvcrt.locking. All three are called locks; none of them block the same set of things.

TL;DR: two processes, 200 read-modify-write cycles each, 400 expected. No lock gave 299. A threading.Lock gave 300. msvcrt.locking gave 400. All three exited with code 0. Four more things bite: msvcrt.locking starts at the current file position, so a file opened in ab mode locks past the end; the lock covers a byte range and not the file; another process reading a locked range gets PermissionError instead of stale data; and os.replace raises WinError 5 while any handle has the target open.

LockScopeCan another process bypass itMeasured here
threading.Lockone processyes, entirely100 of 400 writes lost
msvcrt.lockingbyte range, cross-processno, reads and writes both blocked400 of 400
open(..., 'w')none, no exclusivity on Windowsyestwo handles, last writer wins
`O_CREAT \O_EXCL` lock filepath, cross-processno, but survives a crashnext process blocked forever
os.replaceno lock semanticsn/afails while the target is open

Experiment one: the 101 missing writes

The counter is a fixed-width 8-byte number. Each worker repeats 200 times: read it, sleep 0.5 ms, write it back. Two workers run in parallel.

mode=nolock workers=2 iterations_each=200 expected=400 actual=299 lost=101 exit_codes=[0, 0] wall=1.70s
mode=tlock  workers=2 iterations_each=200 expected=400 actual=300 lost=100 exit_codes=[0, 0] wall=1.71s
mode=flock  workers=2 iterations_each=200 expected=400 actual=400 lost=0   exit_codes=[0, 0] wall=2.35s

Look at the tlock row. I actually wrapped the whole read-modify-write inside with guard:, and the guard only queues threads inside its own process. Each process holds its own lock and neither looks at the other, so the loss is the same as with no lock at all: 100 against 101, and the one-write difference is scheduling noise rather than a lock doing its job.

With the byte-range lock, 400 out of 400, at the cost of wall clock going from 1.70 s to 2.35 s. That extra 0.65 s is real cross-process queueing.

The part worth keeping: all three modes exited 0. A lost update never turns anything red. It just lets a number drift.

Experiment two: the lock starts at the file pointer, not the start of the file

The documentation says it plainly: "The locked region of the file extends from the current file position for nbytes bytes, and may continue beyond the end of the file." The first half is the trap. From the current position, not from byte zero.

I wrote 64 bytes, let another process lock bytes 0 through 7, then opened the same file in ab mode from a second process and called msvcrt.locking:

Another process holds 0..7 · this process opens with 'ab' then locks nbytes=8
Result: success. The lock sits at 64..71, nowhere near the conflict

ab puts the pointer at end of file, so the lock landed on bytes 64 through 71. Both processes believed they held a lock and neither raised anything. That is the shape of "locked but not locked", and it leaves no log line behind.

The fix is to f.seek(offset) before locking, and to seek to the same position before unlocking, because unlock is also relative to the current pointer.

Experiment three: the range is bytes, not the file

With 0 through 7 locked, reading bytes 8 through 15 works fine:

0..7 locked, reading 8..15: b'xxxxxxxx'

Lock a range that partially overlaps and it fails immediately:

holder holds 0..7 · this process seeks to 4 and locks 8 bytes, covering 4..11
PermissionError: [Errno 13] Permission denied

The documentation is explicit on both points: "Multiple regions in a file may be locked at the same time, but may not overlap. Adjacent regions are not merged; they must be unlocked individually." Adjacent regions are never merged, so locking 8 bytes and unlocking 10 does not unlock 8. It raises.

My own first script died exactly there:

lock 8 bytes, then seek(0) and unlock 16 bytes
PermissionError: [Errno 13] Permission denied

The 8-byte lock was still held afterwards, because the exception skipped f.close(). The next four cases in the same process failed one after another, and for a while I believed "locking beyond end of file raises". It does not: run in isolation, locking 4096 bytes on a 16-byte file succeeds, and the docs say so.

Experiment four: on Windows the byte lock is mandatory, other processes cannot even read

This is where Windows and Linux part company. With another process holding bytes 0 through 7, I used the most ordinary read there is:

open(path, 'rb').read(8)
PermissionError: [Errno 13] Permission denied

Not a stale value. No access at all. Contrast flock(2): "flock() places advisory locks only; given suitable permissions on a file, a process is free to ignore the use of flock() and perform I/O on the file." On Linux you have to opt in to checking. On Windows, LockFile is a different contract: "Locking a region of a file gives the threads of the locking process exclusive access to the specified region using this file handle."

The two worlds meet over SMB. The same manual page says, in the CIFS section, "the locks are not advisory anymore: any IO on a locked file will always fail with EACCES when done from a separate file descriptor." Code that works on a local Linux disk because nothing checks the lock will start failing on a network share, with the same error shape as Windows.

A second handle inside the same process is blocked too, documented and measured:

f1 locks 0..7 in this process, f2 = open(path, 'r+b') reads 8 bytes
PermissionError: [Errno 13] Permission denied

Microsoft's wording: "If the locking process opens the file a second time, it cannot access the specified region through this second handle until it unlocks the region." So do not assume that the process holding the lock can freely read through a different handle.

Experiment five: a crash releases the byte lock and leaves the lock file behind

The two designs behave very differently on the same machine.

Byte lock: the holder calls os._exit(9), skipping every cleanup, and the next process takes the lock immediately, measured at 0.000 s.

Holder called os._exit(9); another process locked the same range in 0.000s

Microsoft explains why: "If a process terminates with a portion of a file locked or closes a file that has outstanding locks, the locks are unlocked by the operating system." It also warns that the time this takes depends on available system resources, so explicit unlock on shutdown still matters.

A lock file has no such fallback. O_CREAT | O_EXCL blocks the second process, which is the intent:

second process: os.open(lock, O_CREAT|O_EXCL|O_RDWR)
FileExistsError: [Errno 17] File exists

Then the crash:

lock file still present after os._exit(9): True
next process still blocked: FileExistsError [Errno 17] File exists

The file does not disappear because its creator died, so the job never starts again until somebody deletes it by hand. That is why a lock file should carry a PID and a start timestamp and get cleaned up on boot, and why a bare PID is not enough: PIDs get reused.

Experiment six: atomic replace and locks fight each other on Windows

Temp file plus os.replace is the standard way to write a config, and on Windows it has a precondition: nothing may have the target open.

Target open via an ordinary 'rb' handle, temp file written, then os.replace
PermissionError: [WinError 5] Access is denied: 'lab3.tmp' -> 'lab3.dat'

The reason is sharing mode. Microsoft's CreateFile page says FILE_SHARE_DELETE "Enables subsequent open operations on a file or device to request delete access ... Delete access allows both delete and rename operations." A handle opened without that flag blocks every later delete or rename, and replacement is exactly delete-then-rename.

The worse combination is a byte lock. With another process holding a lock on the target, replacement fails the same way. I ran both cases and only the lock-free, handle-free one succeeded:

byte lock held:  os.replace failed -> PermissionError [WinError 5]
no lock, no handle: os.replace succeeded

One pure syntax difference as well: os.rename never gets as far as the lock, because the target exists. FileExistsError: [WinError 183] Cannot create a file when that file already exists. Use os.replace to overwrite, or check first if you want to preserve the old file.

That gives an ordering rule: on Windows, locking a file and then atomically replacing it is self-contradictory, because the lock itself makes the replace fail. Either lock a separate small lock file and replace the data file, or keep the lock and write in place.

Experiment seven: appends are atomic, as long as they are small

Two processes each appended 500 lines of 5 bytes through ab with buffering=0:

500 lines each from two processes · 1000 lines on disk · 0 torn lines

That one has an edge, so do not generalise. I measured one complete 5-byte record per write call with nothing split. Longer lines, or a multi-megabyte write, can behave differently once CRT buffering and the filesystem get involved. For anything large, go back to temp file plus replace.

Six real errors and what to do with each

All six were captured on this machine, ordered by how often they show up.

Error 1: PermissionError: [Errno 13] Permission denied from msvcrt.locking.

Three causes: no seek, so the lock landed elsewhere; partial overlap with an existing lock; unlock length that does not match the lock. Fix: f.seek(offset) before locking, seek again before unlocking, and keep the length in one variable shared by both calls.

Error 2: OSError: [Errno 36] Resource deadlock avoided from LK_LOCK.

That is the blocking variant. The docs say "If the bytes cannot be locked, the program immediately tries again after 1 second. If, after 10 attempts, the bytes cannot be locked, OSError is raised." Measured wait: 9.07 s. Note the wording is not "timeout" but "deadlock avoided", so do not log it as a timeout or the next person will go looking at timeout settings.

Error 3: PermissionError: [WinError 5] Access is denied from os.replace.

The target is open, or its bytes are locked. Fix: make sure no handle holds the target when replacing, or move the lock to a separate file. When this hit me, a read handle had escaped its with block, and the message gives you only two paths and no hint about who is holding it.

Error 4: FileExistsError: [WinError 183] Cannot create a file when that file already exists from os.rename.

os.rename does not overwrite on Windows. Switch to os.replace. This one often arrives right after the previous fix, because the target may still be open.

Error 5: FileExistsError: [Errno 17] File exists from O_CREAT | O_EXCL.

Either a real second instance is running, or a crashed one left the lock file. Judge by the PID and start time inside the file, not by deleting it.

Error 6: ModuleNotFoundError: No module named 'fcntl'

fcntl is a Unix module. Cross-platform scripts that put import fcntl at module level do not fail at lock time; they fail to start on Windows at all. Import per platform instead.

A minimal version that holds

This encodes the three findings: seek before locking, one length variable, seek again before unlocking.

import contextlib, msvcrt, time


class ByteLock:
    def __init__(self, path, offset=0, length=1, timeout=5.0):
        self.path = path
        self.offset = offset
        self.length = length
        self.timeout = timeout

    @contextlib.contextmanager
    def hold(self):
        f = open(self.path, 'r+b')
        try:
            for _ in range(max(1, int(self.timeout / 0.05))):
                f.seek(self.offset)
                try:
                    msvcrt.locking(f.fileno(), msvcrt.LK_NBLCK, self.length)
                    break
                except PermissionError:
                    time.sleep(0.05)
            else:
                raise TimeoutError(
                    'lock busy: %s [%d:%d]' % (self.path, self.offset, self.length))
            try:
                yield f
            finally:
                f.seek(self.offset)
                msvcrt.locking(f.fileno(), msvcrt.LK_UNLCK, self.length)
        finally:
            f.close()

Use it as with ByteLock('state.dat', 0, 8).hold() as f:. Three notes. LK_NBLCK with your own polling instead of LK_LOCK keeps the timeout yours and gives you a TimeoutError you can read. PermissionError must be caught before OSError, since it is a subclass, or the broader handler swallows it. And f.close() sits in the outermost finally so the handle is released even if the unlock fails; otherwise the lock survives until the process dies.

The same code is a different contract on Linux

Three differences, none of which requires changing a line.

First, fcntl does not exist on Windows and msvcrt does not exist on Linux, so a top-level import breaks one side or the other at startup.

Second, flock on Linux is advisory: anyone who does not check can write straight through. Windows byte locks are mandatory: another process cannot even read. So "nobody else will write once I hold the lock" is true on this Windows box and false on Linux.

Third, release timing differs. flock(2) binds the lock to the open file description, and "the lock is released either by an explicit LOCK_UN operation on any of these duplicate file descriptors, or when all such file descriptors have been closed". On Windows the lock follows the handle, and a second handle in the same process cannot read its own locked range. Both models work; they are not interchangeable.

If the code must run on both, branch at the lock layer (Windows on msvcrt, everything else on fcntl) or use a maintained cross-platform lock library. The two details hand-rolled abstractions always miss are the unlock length and the second handle.

What I did not test

There is no Linux box here, so the POSIX side is documentation only, with no measured numbers. Network filesystems (a real SMB or NFS mount) were not exercised; I only quote the CIFS paragraph from the manual page. Mixed threads and processes, appends larger than the buffer, and os.replace across volumes are all uncovered.

The usable conclusion: on a local Windows disk, a byte-range lock is the only option I measured that actually blocks another process, and it covers a byte range, respects the current file pointer, and only protects the file object that is open at the time. To cover a whole file, lock a length that spans it explicitly, and re-acquire after the file is replaced.

This is the same theme as the Windows path semantics piece: the string you write, the handle you open, and the object you think you have are often three different things. To go further, read why an import loads the wrong file and why a hot SQLite backup opened fine with 14,500 rows missing. The first is about which file a module object points at, the second about a backup that opens and is still short of data, and both belong to the same family of silent divergence.

Questions I get asked

Q: Both processes used threading.Lock, so why did writes still go missing?

Because a threading.Lock is an object inside one process. Two processes have two of them and neither can see the other. Cross-process exclusion needs an operating-system lock: msvcrt.locking, fcntl.flock, or a lock file. Measured here: 100 writes lost out of 400, essentially the same as no lock.

Q: Is Resource deadlock avoided from LK_LOCK a timeout?

No. It is ten attempts one second apart, then give up, measured at 9.07 s. The errno is 36 and the text says deadlock, not timeout. For your own timeout, use LK_NBLCK and poll.

Q: Can I just lock the whole file?

Byte locks have no "whole file" unit, only a range. To cover everything, lock a length that spans the file; locking past the end is allowed and measured (4096 bytes on a 16-byte file). Remember that once os.replace swaps the file, the lock is still on the old object and the new file is unlocked.

Q: Is a lock file simpler?

Simpler, but it survives a crash and then blocks every future run, and PID reuse makes liveness checks unreliable. A byte lock is released by the system when the process dies; measured at 0.000 s for the next process. Pick based on which failure you would rather debug: locking nothing, or never starting.

Three checks to paste into a code review

One: anywhere you see cross-process exclusion, search for threading.Lock and multiprocessing.Lock used in the wrong place. The first one only queues threads.

Two: wherever msvcrt.locking appears, check that lock and unlock both seek to the same offset and share one length variable.

Three: for every temp-file-plus-os.replace, check whether anything still holds the target open. On Windows that is a hard precondition, not a suggestion.

👉 Join Xiaomi MiMo Platform: Leading AI model platform with cost-effective inference

👉 Join Aliyun AI: Top AI products with exclusive coupons for business innovation

📌 This article was AI-assisted generated and human-reviewed | TechPassive — An AI-driven content testing site focused on real tool reviews

🔗 Recommended Tools

These are carefully selected tools. Using our affiliate links supports us to keep producing quality content:

As an Amazon Associate, I earn from qualifying purchases at no extra cost to you.

☁️ DigitalOcean Cloud ⚡ Vultr VPS 🤖 QoderWork CN (Refer & Earn) ☁️ Aliyun AI Products 📚 WordPress Books 🧠 AI & LLM Books 🏗️ System Design Books 🐳 Docker & K8s Books 🐧 Linux Admin Books 🐍 Python Books 🗄️ Database Books 🚀 DevOps & SRE Books 🤖 Xiaomi MiMo Platform
← Back to Home