Linux flock in Practice: Concurrency Safety for AI Pipelines and 3 Real Pitfalls

flockconcurrencyAI pipelineDevOpsLinuxpractice2026

This post walks through 3 real production pitfalls and shows the correct usage of flock(2), flock -xn, and atomic mv replacements in an AI pipeline.

Why AI automation pipelines fear concurrency more than traditional cron

Traditional cron jobs are usually single commands, serial execution, write one file then move on. AI automation pipelines are different—they have at least three concurrency sources:

1. **Multiple cron slots competing for the same file** (6AM/12PM/18PM/22PM all writing /tmp/article-gen/cn.txt)

2. Isolated session scheduling drift (22PM started at 21:42, 12PM leftover subagent still running at 21:47)

3. **Read-during-write window between pipeline phases** (Phase 0 reads drafts/cn.txt, Phase 2's generate-html.py reads /tmp/article-gen/*.txt, 5-8 second window in between)

The 7/19 incident concrete replay: the 22PM session's 12 push commits were all 7/19 archive content, 0 of them were 22PM new articles. I thought it was a model call bug, but after a night of debugging I realized /tmp/article-gen/cn.txt was overwritten by 3 sessions in 5 seconds, last writer won, the 12PM subagent that started at 21:47 entirely replaced the 22PM topic.

⏳ TL;DR

🥇 **Core mechanism**: flock(2) file lock + -xn (non-blocking exclusive) mode + mv atomic replacement combo

👉 View util-linux flock docs

🌟 **Key fix**: use isolated /tmp/article-gen/cn-{pid}.txt filenames + wrap write-and-run in flock -xn /tmp/article-gen.lock

👉 View Linux man flock(1)

💻 **Defense command**: cp new-cn.txt /tmp/article-gen/cn.txt.tmp && mv /tmp/article-gen/cn.txt.tmp /tmp/article-gen/cn.txt (read-during-write defense)

👉 View generate-html.py v3 case

🛠️ Prerequisites

**System**: Ubuntu 22.04 LTS / 24.04 LTS (other Linux distros work too, macOS `flock` comes from util-linux port with different install)
**Kernel**: Linux 5.15+ (`flock(2)` stable since 1990s, no version requirements)
**Dependencies**: `util-linux` (preinstalled, almost every distro ships it)
**Verify command**:

which flock && flock --version | head -1
# util-linux 2.37.2 (Ubuntu 22.04) / 2.39.x (Ubuntu 24.04)

💣 3 Real Production Pitfalls

Pitfall 1: `flock` without `-x` flag, 6 sessions got 6 shared locks

My first implementation:

flock /tmp/article-gen.lock -c "echo hello"

This looked correct, but flock defaults to **shared lock**—multiple processes can hold shared locks simultaneously. All 6 cron sessions got the lock, all entered the critical section, all wrote /tmp/article-gen/cn.txt at the same time. **The lock became an "I'm writing" broadcast, providing zero mutual exclusion**.

Fix:

# -x = exclusive lock (mutual exclusion)
# -n = non-blocking fail immediately instead of waiting
flock -xn /tmp/article-gen.lock -c "python3 /root/.openclaw/workspace/affiliate-blog/run-pipeline.py"

-xn double flag: of 6 concurrent processes, only 1 immediately gets the lock, the other 5 fail immediately (exit code 1). The lock holder exclusively runs the entire pipeline until done. Verify:

flock -xn /tmp/test.lock -c "sleep 5" &
flock -xn /tmp/test.lock -c "echo got it" ; echo "exit: $?"
# exit: 1   ← second flock failed immediately, didn't wait 5 seconds

Pitfall 2: Filename without PID, got overwritten by 6AM leftover subagent

The 7/20 incident root cause: 22PM wrote /tmp/article-gen/cn.txt (AI pipeline v3 flock topic), but 6AM 6:12 leftover subagent wrote /tmp/article-gen/cn.txt (WP 7.0 Playground topic), **both sessions used the same filename**. Last writer won.

Fix: each session uses isolated filename with PID + cron tag:

SESSION_TAG="22pm-cron"
PID=$$
INPUT_FILE="/tmp/article-gen/cn-${SESSION_TAG}-${PID}.txt"

cp drafts/cn.txt "$INPUT_FILE"
flock -xn /tmp/article-gen.lock -c "python3 generate-html.py --input-file $INPUT_FILE"

The `--input-file` flag was added in the 7/20 upgrade, so `generate-html.py` no longer hardcodes the line `/tmp/article-gen/cn.txt`. Full v3 implementation at yaohehe/yaohehe.github.io `generate-html.py` lines 247-263.

Pitfall 3: cp + rm replacement is not atomic, read-during-write gets half files

My early "fix":

cp new-cn.txt /tmp/article-gen/cn.txt
rm /tmp/article-gen/cn.txt.bak

The problem: Phase 2's generate-html.py opened the file while Phase 1's cp was still writing, read half new content half old content, generated HTML with half yesterday's topic half today's topic. **Linux cp(1) opens target → writes → closes, during write the target is readable by other processes**.

Fix: write to temp file + mv atomic replacement. mv within the same filesystem is atomic (rename(2) system call guarantee), the "half-written file" state never appears:

cp new-cn.txt /tmp/article-gen/cn.txt.tmp
mv /tmp/article-gen/cn.txt.tmp /tmp/article-gen/cn.txt
# At any moment, Phase 2 either sees the old file or the complete new file

Why mv is atomic: rename(2) in POSIX spec is "atomic inode pointer replacement"—the kernel switches directory entry pointers at the directory level, either the switch hasn't happened (old file visible) or it's complete (new file visible), no intermediate state. cp is open + write + close three steps, fully readable throughout.

🚀 Practice: complete diff for upgrading generate-html.py to v3

v2 → v3 key 3 changes:

# generate-html.py v2 (pre 7/20, wrong implementation)
INPUT_FILE = "/tmp/article-gen/cn.txt"
def render():
    with open(INPUT_FILE) as f:
        return f.read()

# generate-html.py v3 (post 7/20, correct implementation)
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--input-file", required=True)
args = parser.parse_args()

def render(input_file):
    with open(input_file) as f:
        return f.read()

# run-pipeline.py v3 wrapper
SESSION_TAG = os.environ.get("CRON_TAG", "manual")
PID = os.getpid()
INPUT_FILE = f"/tmp/article-gen/cn-{SESSION_TAG}-{PID}.txt"
LOCK_FILE = "/tmp/article-gen.lock"

# 1. Isolated filename
shutil.copy("drafts/cn.txt", INPUT_FILE)

# 2. flock -xn exclusive lock + non-blocking
try:
    subprocess.run([
        "flock", "-xn", LOCK_FILE, "-c",
        f"python3 generate-html.py --input-file {INPUT_FILE}"
    ], check=True)
except subprocess.CalledProcessError as e:
    if e.returncode == 1:
        # Lock held by another session, fail immediately without blocking
        logger.warning(f"flock busy, exit 1, skip this run")
        sys.exit(0)
    raise

The --input-file flag changed from v2 hardcode to v3 explicit, generate-html.py no longer assumes file path. This is the core of the 6/23 corrections upgrade to v3—**turn "I assume the file is here" into "I'm told the file is here"**.

🛡️ Advanced: 3 production hardening items

Hardening 1: last_run timestamp detection

Even with flock, add a 10-minute "last run timestamp" check to avoid the concurrent peak at the moment of lock release:

LAST_RUN_FILE="/tmp/article-gen/.last-run"
if [ -f "$LAST_RUN_FILE" ]; then
    LAST_RUN=$(cat "$LAST_RUN_FILE")
    NOW=$(date +%s)
    DIFF=$((NOW - LAST_RUN))
    if [ $DIFF -lt 600 ]; then
        echo "Recent run ${DIFF}s ago, skip"
        exit 0
    fi
fi
date +%s > "$LAST_RUN_FILE"

Hardening 2: log isolation to avoid pollution

LOG_FILE="/tmp/article-gen/logs/cn-${SESSION_TAG}-${PID}.log"
mkdir -p "$(dirname "$LOG_FILE")"
flock -xn /tmp/article-gen.lock -c "python3 ... 2>>$LOG_FILE"

Isolate logs by session tag + PID, so 6AM/12PM/18PM/22PM logs don't mix. During the 7/19 incident I debugged for an hour because logs were all 6 sessions mixed—after adding PID suffix, locate time dropped to 5 minutes.

Hardening 3: monitoring script mount

# Every day 23:00 check today's concurrent failure count
0 23 * * * grep -c "flock busy" /tmp/article-gen/logs/*.log | awk -F: '$2>10 {print $1}' | mail -s "Flock busy spike" admin@example.com

flock busy more than 10 times/day means cron schedules overlap, need to push schedule gap to ≥30 minutes.

Summary and next steps

After 7/20 upgrade to v3, W28-W29 7 days 0 similar path conflicts (corrections.md 2026-07-26 W28 verification record). The flock + isolated filename + atomic mv combo turned the AI automation pipeline from "gamble" to "engineering problem".

Next steps:

1. **flock clustering**: single-machine flock suffices, but multi-VPS pipelines need redis-lock to replace /tmp/article-gen.lock (extends 7/3 Ansible Molecule multi-distro practice)

2. **Generator + pusher dual locks**: publish-via-api.py also adds flock, avoiding 6AM/22PM pushing different commits to main branch simultaneously

3. **observability integration**: pipe flock busy count into Langfuse traces (extends 6/20 n8n + Langfuse v3 article)

flock(2) is a 1990s-stable system call, and in 2026 it's still solving the most basic concurrency problem for AI pipelines. **The plainest tools are often the hardest to replace**.