AI Automation Pipeline Concurrency Conflict Post-Mortem: 5 Real Scenarios and the flock Lock Fix

Last year I built an OpenClaw AI automation pipeline that runs 4 cron tasks × 4 times daily × isolated sessions × generate → render HTML → push to GitHub Pages. It ran cleanly for 5 weeks (W21-W26), then on a single day in W30 (2026-07-20) it tripped into the same trap **3 separate times**: the 6AM cron got overwritten by leftover 22PM state, the 22PM cron got overwritten by other cron subagents, and the four articles I thought I published (Thunderbolt 4 / WordPress 7.0 / NAS NIC / Apple USB-C) actually contained **zero lines from my cn.txt**—three writes, three replacements. This post tears the crash → fix → recrash → final lock down to its bones, focusing on why the v2 "dual-source sync" fix from corrections.md 6/23 was fundamentally wrong, and how v3 with flock -xn /tmp/article-gen.lock plus isolated filenames actually solves it. If you run multiple cron jobs sharing a temp directory, these 5 crash scenarios + flock lock + 8-point production checklist are worth reading end-to-end.

⏳ Too Long; Didn't Read

• **Root cause**: 4 cron tasks (6AM/12PM/18PM/22PM) running in isolated sessions share `/tmp/article-gen/`. Multi-session concurrent `write()` to the same file means **last writer wins**, with a ~30 second window where any other session can swap your content.
• **Symptom**: You write cn.txt successfully (10559 bytes) → Phase 0 reads it OK → Phase 2 generate-html.py reads something else—NAS NIC / Thunderbolt / Apple USB-C written 30 seconds earlier by someone else. Pipeline pushes 12 articles, none are yours.
• **Why v2 fix (6/23 dual-source sync) was insufficient**: We thought the problem was "drafts/ vs /tmp/article-gen/ mismatch". Actually the problem is "multi-session overwriting a single source"—you can't sync two copies of a file that's already been hijacked.
• **The v3 fix that works**: `flock -xn /tmp/article-gen.lock` wrapping the entire write-then-run sequence + isolated filenames like `${STAMP}-cn.txt` (cron name + timestamp) + `generate-html.py --input-file ` argument + atomic `mv` replacement.

Why /tmp/article-gen/ Became a Multi-Session Battlefield

The pipeline design has run-pipeline.py Phase 1 calling generate-html.py, which hardcodes reading /tmp/article-gen/cn.txt and /tmp/article-gen/en.txt (not drafts/cn.txt). Simple reasoning: tmpfs is faster than the working directory, and no relative-path resolution across sessions.

But the W21+ scheduling policy changed: the 6AM cron doesn't fire at a fixed time—it depends on subagent queue depth. When 6AM is still writing cn.txt, 22PM (actually triggering at 21:53) is already starting run-pipeline.py. Both share the filename /tmp/article-gen/cn.txt:

Timeline (2026-07-20, real)
21:42  22PM isolated session starts, writes /tmp/article-gen/cn.txt (10559 bytes)
21:44  6AM isolated session starts, overwrites cn.txt (14752 bytes, NAS NIC topic)
21:47  6AM Phase 0 reads cn.txt, sees topic changed to NAS NIC → rewrites
21:48  6AM rewrite gets overwritten again by 22PM (22PM is generating HTML now)
21:50  6AM waits for 22PM last_run to complete, rewrites (16752 bytes WordPress 7.0)
21:51  6AM pipeline succeeds, publishes 4 articles (this CN+EN pair + 10 drafts-queue historicals)
21:53  22PM also finishes, but the "AI Automation Pipeline 5 Bugs" topic it wrote produced 0 pushed articles

This is the case v2 fix (6/23 dual-source sync) cannot handle: your cn.txt is being overwritten by someone else—not a consistency problem, an ownership problem.

💣 Crash Log: 5 Real Scenarios (Reverse Chronological)

Scenario 1: 22PM 21:53 Third Trip into the Same Hole (Worst, 2026-07-20)

Symptom: 22PM isolated session writes cn.txt at 21:42 ("AI Automation Pipeline 5 Bugs" topic) → pipeline runs at 21:46 → Phase 1 generate-html.py reads cn.txt and finds NAS NIC topic (overwritten by 6AM at 21:44) → 12 generated articles are all from 6AM topic or drafts-queue historicals, 0 articles from 22PM's "5 Bugs" content.

**Root cause**: run-pipeline.py Phase 0 reads /tmp/article-gen/cn.txt, then there's a **30-second window** before Phase 1 generate-html.py reads it again. During that window, the 6AM isolated session overwrites the file.

Fix: v3 flock lock (see below)—the entire write-then-run sequence must be inside one lock, no other session allowed to intervene.

Scenario 2: 6AM and 22PM Overwriting Each Other (21:48, 2026-07-20)

Symptom: 6AM starts writing cn.txt at 21:44 (WP 7.0 Playground topic, 14752 bytes), finishes at 21:46 → but 22PM cron starts run-pipeline.py at 21:47, overwrites cn.txt with NAS NIC topic → 6AM Phase 0 reads and sees "topic was changed", rewrites at 21:48 → gets overwritten again by 22PM.

Root cause: Two cron sessions have zero coordination. Each assumes it's the unique writer. When 6AM waits 30s to retry, 22PM is mid-Phase-2, overwriting again.

Fix: v3 flock lock + ≥30 minute scheduling offset (already promoted to permanent standard #24).

Scenario 3: v2 "Dual-Source Sync" Fundamentally Doesn't Work (2026-06-23)

Symptom: 6/23 22PM cron trips into the same hole for the second time, generate-html.py path vs drafts/ path mismatch.

Fix attempted: v2 fix—copy drafts/cn.txt to /tmp/article-gen/cn.txt before running pipeline.

Why it failed: v2 assumed "drafts/ is the source of truth, /tmp/article-gen/ is a temp copy". But in the 7/20 case, drafts/cn.txt was 22PM's "5 Bugs", while /tmp/article-gen/cn.txt had already been swapped to NAS NIC by 6AM. Phase 0 reading drafts/cn.txt is correct, but Phase 1 generate-html.py reads /tmp/article-gen/cn.txt (wrong), and generate-html.py has no parameter to point at drafts/.

Lesson: v2 only solved "drafts/ stale pollution of /tmp/article-gen/". It does nothing for "multi-session overwriting the same file."

Scenario 4: 6AM 6:13 Stale Pollution (2026-07-17, W29)

Symptom: After 6AM cron completes, /tmp/article-gen/cn.txt remains (Ansible topic, 06:13 mtime). That evening 22PM cron starts run-pipeline.py, generate-html.py reads Ansible topic, the WordPress articles 22PM wrote produce 0 pushed articles.

**Fix attempted**: Manually rm /tmp/article-gen/*.txt or mv to backup after 6AM.

Lesson: v2 addressed the "single stale residual" case. v3 must extend to "concurrent overwrite" cases.

Scenario 5: Cron Subagent Queue Unintended Writes (2026-07-20)

Symptom: After 22PM cron starts, the 6AM subagent queue still has 12PM/18PM subagents that didn't complete. These subagents, at 22PM's 21:47, write to /tmp/article-gen/. Nobody told them to—it's the scheduler's auto-recovery mechanism.

Root cause: OpenClaw's scheduler resumes all unfinished subagents when a cron triggers (default 1-hour window). If 12PM's subagent was incomplete, it auto-recovers and runs remaining tasks during 22PM's session, possibly including writes to /tmp/article-gen/.

Fix: v3 flock lock + scheduling offset ≥30 minutes (avoid subagent recovery window overlap).

🛠️ The Real Fix: v3 flock Lock + Isolated Filenames (Complete Script)

Step 1: Write to Isolated Filenames (Cron Name + Timestamp)

#!/bin/bash
# 22PM cron header: pick isolated filenames first
STAMP="$(date +%Y-%m-%d-%H%M)-22pm"  # e.g., 2026-07-20-2153-22pm
CN_FILE="/tmp/article-gen/${STAMP}-cn.txt"
EN_FILE="/tmp/article-gen/${STAMP}-en.txt"

# Check whether other crons are running (v3 permanent standard #24)
if ps auxf | grep -E "publish-via-api|run-pipeline|generate-html" | grep -v grep > /dev/null; then
  echo "⚠️ Another cron pipeline is running, waiting 30s and rechecking"
  sleep 30
  if ps auxf | grep -E "publish-via-api|run-pipeline|generate-html" | grep -v grep > /dev/null; then
    echo "❌ Active process still present, aborting this publish"
    exit 1
  fi
fi

# Wrap the entire write-run sequence in flock (the key!)
exec 9>/tmp/article-gen.lock
if ! flock -xn 9; then
  echo "⚠️ flock lock held by another cron, aborting"
  exit 1
fi

# Inside the lock, write the files
cat > "$CN_FILE" <<'EOF'
# Your CN article content (pipe-format metadata + body)
EOF

cat > "$EN_FILE" <<'EOF'
# Your EN translated version
EOF

# Phase 1: call generate-html.py, **specifying input file** (v3 new arg)
python3 generate-html.py --cn "$CN_FILE" --en "$EN_FILE" \
  --out-cn /root/.openclaw/workspace/yaohehe.github.io/2026-07-20-xxx.html \
  --out-en /root/.openclaw/workspace/yaohehe.github.io/2026-07-20-xxx-en.html

# Phase 2: push (still inside the lock)
python3 /root/.openclaw/workspace/yaohehe.github.io/publish-via-api.py 2026-07-20-xxx.html
python3 /root/.openclaw/workspace/yaohehe.github.io/publish-via-api.py 2026-07-20-xxx-en.html

# Cleanup (keep 24h for debugging)
find /tmp/article-gen/ -name "${STAMP}-*" -mmin +1440 -delete

# Lock auto-releases on script exit

Step 2: Add --input-file Arguments to generate-html.py

# generate-html.py patch
import argparse

parser = argparse.ArgumentParser()
parser.add_argument('--cn', default='/tmp/article-gen/cn.txt',
                    help='CN input file path (v3 new argument)')
parser.add_argument('--en', default='/tmp/article-gen/en.txt',
                    help='EN input file path (v3 new argument)')
parser.add_argument('--out-cn', required=True, help='CN output HTML path')
parser.add_argument('--out-en', required=True, help='EN output HTML path')
args = parser.parse_args()

# Read v3-specified input files (no longer hardcoded)
with open(args.cn, 'r', encoding='utf-8') as f:
    cn_content = f.read()
with open(args.en, 'r', encoding='utf-8') as f:
    en_content = f.read()

# ... rest of HTML generation unchanged

Step 3: Scheduling Offset Check (Permanent Standard #24)

# Before 22PM cron triggers, check if any cron completed in the last 30 minutes
LAST_RUN=$(stat -c '%Y' /root/.openclaw/workspace/affiliate-blog/drafts/.last_run 2>/dev/null || echo 0)
NOW=$(date +%s)
DIFF=$((NOW - LAST_RUN))

if [ "$DIFF" -lt 1800 ]; then
  echo "⚠️ Only ${DIFF}s since last cron completion, waiting $((1800 - DIFF))s"
  sleep $((1800 - DIFF))
fi

🔒 Why flock Is the Correct Fix (Not atomic mv)

A common first reaction: "Can't we just use mv for atomic replacement?"—**No.**

The key advantage of flock -xn: **non-blocking semantics**—if the lock is held, fail immediately, don't make cron wait 5 minutes. Otherwise two crons waiting on each other's locks produce **near-deadlock** (mutual 30-second timeout, but during those 30 seconds neither can work).

🛡️ 8-Point Production Verification Checklist

After upgrading to v3, you must verify:

• [ ] **V1**: When two crons trigger simultaneously, the second `flock -xn` aborts immediately (no blocking)
• [ ] **V2**: After lock release, a retried cron can write files normally
• [ ] **V3**: `generate-html.py --cn /tmp/article-gen/22pm-xxx-cn.txt` reads the specified file, not the default cn.txt
• [ ] **V4**: HTML mtime pushed by publish-via-api.py is after the file write (not before overwriting)
• [ ] **V5**: 6AM/22PM scheduling offset ≥30 minutes (subagent recovery windows don't overlap)
• [ ] **V6**: `/tmp/article-gen.lock` file exists with permissions 0644
• [ ] **V7**: Files in `/tmp/article-gen/` older than 24 hours are auto-cleaned (`find -mmin +1440 -delete`)
• [ ] **V8**: 7 consecutive days × 4 crons × 4 runs = 28 publishes, zero topic misfires

🔗 Related Internal Links

• 6/23 22PM cron second trip into the same hole (v2 fix root-cause insufficient): corrections.md 6/23 entry
• 7/17 6AM stale pollution (origin of v3 flock lock idea): corrections.md 7/17 entry
• 7/20 21:48 / 21:53 dual-crash case (core data source for this article): corrections.md 7/20 entry

Summary and Next Steps

AI automation pipeline concurrency conflicts are not "dual-source inconsistency" issues—they're "multi-session ownership of a single file" issues. v3 needs flock -xn + isolated filenames + generate-html.py --input-file parameter, all three stacked, to fully solve it. **W30 (this week) is already promoted to permanent standard #24**, and all 4 cron task (6AM/12PM/18PM/22PM) prompts must hardcode this flow in their headers.

Next steps:

1. W31 verification: 7 consecutive days observing whether the 4 crons still hit similar conflicts (V8 checklist)

2. run-pipeline.py v2 upgrade: Phase 0 auto-wraps in flock lock, cron prompts don't need manual locking

3. Subagent scheduling window adjustment: shorten OpenClaw scheduler's subagent recovery window from 1 hour to 30 minutes

👉 Join MiniMax Token Plan: AI coding acceleration for businesses

👉 Join Zhipu Coding Plan: GLM-4.6/GLM-5 coding packages, China-stable, pay-per-token unlimited

👉 Join Aliyun AI: Top AI products with exclusive coupons for business innovation

📌 This article was AI-assisted generated and human-reviewed | TechPassive — An AI-driven content testing site focused on real tool reviews