AI Automation Blog git pull --rebase 177 Articles 404 Production Incident Postmortem 2026: Permanent Rule #24 Origin and 3-Layer Defense

I am a programmer and I run a cron + AI Agent pipeline that auto-publishes articles to my blog. On June 4 2026 at 12:17 PM Asia/Shanghai, my GitHub Pages site (yaohehe.github.io) suddenly 404'd a large batch of published articles — all 177 articles published between May 19 and June 4 became 404, the homepage, search traffic, and Clarity heatmap dropped to zero instantly.

I traced the root cause through git reflog: the 12PM cron ran git pull --rebase origin main at 12:08 (using the remote 5/31 old version as rebase base), then the 18PM pipeline force-pushed the remote from the complete 5/19~6/4 version back to the 5/19-missing old version. Worse, my manual ceecb9bf force push at 11:00 had already committed all 706 files into main, but the 12PM task's 12:08 rebase erased the 5/19~6/3 content from the commit history.

I recovered everything within 2 minutes using reflog + working tree (git reset --hard 1ef91bbf + force push), but this incident led to **Permanent Rule #24** — all cron pipelines are now banned from any git pull/rebase/push operation. This article is the complete **postmortem of that real production incident**, including root cause anatomy, rescue commands, and 3-layer defense mechanism deployment checklist.

Incident Timeline (to the Minute)

Key observation: reflog was still alive, all 706 files were intact locally; only the remote main branch pointer was wrongly rebased to an old position.

Root Cause Anatomy (3 Layers of Stacked Mistakes)

Root Cause 1: 12PM Cron Ran `git pull --rebase` Without Protection

The original 12PM cron prompt contained a "keep local and remote in sync" logic, and the AI implementation directly invoked git pull --rebase origin main:

# ❌ Dangerous code (original 12PM implementation)
cd /root/.openclaw/workspace/yaohehe.github.io
git fetch origin
git pull --rebase origin main    # ← FATAL: use remote as base, replay local commits
python3 generate-html.py
git add .
git commit -m "auto: 12PM batch"
git push                            # ← If push rejected, may force push

Problem: after rebase, the local main branch's commit chain was rewritten. If the remote had new commits (a force push I didn't see), the rebase would "lose" local-only commits — that's exactly the mechanism that wiped 5/19~6/3 article content from the remote main tree in this incident.

Root Cause 2: My 11:00 AM Force Push Had Already Committed 706 Files Into main

I manually force-pushed a large update (706 files), putting all 5/19~6/4 content onto the remote. But the 12PM cron ran rebase again at 12:08 — the rebase used the remote 5/31 old version as base, which essentially rewrote my complete 706-file commit chain using the old chain.

# Remote main chain before incident
A---B---C---D---ceecb9bf (force push, 706 files)
                              \
                               E---F (12PM additions)
# After rebase
A---B---C---D (remote 5/31 old version)
            \
             E'---F' (12PM additions but using old D as base)
# E'/F' have same content as original E/F, but all commits before D are gone

Root Cause 3: 18PM Pipeline Force Push Sent the Rolled-Back main to Remote

The 18PM cron used git push --force-with-lease, which after rebase pushed the current main pointer (pointing to old D) to remote — the remote main tree's 706-file state was flattened back to the 5/19-missing version.

Three layers stacked: manual force push → 12PM erroneous rebase → 18PM re-force-push, erasing all 5/19~6/4 content from the remote.

Rescue Commands (2-Minute Recovery)

I found the last correct commit hash 1ef91bbf from git reflog (i.e., after ceecb9bf's force push) and recovered with these commands:

# Step 1: Check reflog for the correct commit
git reflog | grep -E "ceecb9bf|1ef91bbf"
# Output (example):
# 1ef91bbf HEAD@{0}: commit: 706 files
# a3c4d5e6 HEAD@{1}: rebase finished

# Step 2: Hard reset to the correct commit
git reset --hard 1ef91bbf

# Step 3: Force push
git push --force-with-lease origin main

# Step 4: Verify GitHub Pages
curl -I https://yaohehe.github.io/2026-05-29-claude-code-...-en.html
# HTTP/2 200 ✅

**Key observation**: reflog is retained by default for 90 days. All commits remain local as long as you don't run git gc --prune=now.

Permanent Rule #24 Upgrade: 3-Layer Defense Mechanism

After the incident I established Permanent Rule #24, completely banning git operations in cron pipelines. 3 layers of protection:

Defense Layer 1: All Cron Prompt Headers Get ⚠️ Git Operation Ban

• git pull / git pull --rebase / git pull --no-rebase / git fetch origin
• git push / git push --force / git push --force-with-lease
• git rebase / git reset --hard

• Read-only: git status / git log / git diff / git ls-files
• Push single file: python3 publish-via-api.py (GitHub Contents API, no commit)

# ✅ New rule (solidified in all 6AM/12PM/18PM/22PM cron prompt headers)
⚠️ BANNED git commands (in pipeline, 4AM health-check only):

✅ ONLY allowed:

Why not delete git operations entirely: 4AM health check and manual debugging occasionally need git log / git diff; only destructive commands are banned.

Defense Layer 2: 4AM Health Check Adds "Remote vs Local File Count Comparison"

I added a new check (Step 3 in 4am-health-check.py main()):

def check_remote_vs_local():
    """Compare GitHub remote main file count vs drafts/ published file count"""
    # Fetch remote file list (lightweight API call)
    remote_files = fetch_github_tree("yaohehe", "yaohehe.github.io", "main")
    local_published = load_pushed_manifest()  # .pushed_manifest.json

    diff = len(remote_files) - len(local_published)
    if abs(diff) > 10:  # threshold 10
        send_alert(f"⚠️ Remote vs local file count diff {diff}, possible incident!")
        pause_all_cron_pipelines()  # pause push tasks

Why threshold 10: A single batch typically pushes 2~6 articles, so 10 is a safe margin.

Defense Layer 3: `.pushed_manifest.json` Push Records + 4AM Verification

.pushed_manifest.json records every successful push:

{
  "schema_version": 1,
  "pushed": [
    {
      "path": "2026-05-29-claude-code-...-en.html",
      "timestamp": "2026-05-29T10:30:00Z",
      "commit_sha": "abc1234",
      "size": 32783
    }
  ]
}

The 4AM verification script will:

1. Read .pushed_manifest.json

2. Call GitHub Tree API to get actual remote files

3. Verify all paths exist + mtime matches

4. If missing → alert and recover (re-push via publish-via-api.py)

publish-via-api.py Push Blind Spot (New Discovery)

After the incident I re-audited publish-via-api.py (GitHub Contents API single-file push tool) and discovered **it does not create git commits**. This means:

• Local `git log origin/main..HEAD` shows **no behind** (because there are no new local commits)
• But remote main is actually behind (push via API doesn't write commits)
• Next `git pull --rebase origin main` will "appear synced" but actually conflict

This is the root cause of the dual-track git reference expiry issue (already documented in corrections.md 6/04 11:07).

**Fix**: Add sync_git_refs() step to 4am-health-check.py, running git fetch origin every 4 hours (30s timeout, failure non-blocking) to sync refs:

def sync_git_refs():
    """Sync remote refs to avoid publish-via-api push blind spot"""
    r = run_cmd("git fetch origin", timeout=30, fatal=False)
    if r and r.returncode == 0:
        log("✅ git refs synced")
    else:
        log("⚠️ git fetch failed (non-blocking, possibly network issue)")

5 Permanent Lessons Learned After the Incident

1. **git pull --rebase is a ticking time bomb in automation pipelines**. Never run rebase in cron; sync manually if needed.

2. Force push is a double-edged sword. After manual force-push recovery, you must immediately add protection in the cron pipeline; otherwise the next 12PM task's rebase will wipe your recovered content again.

3. GitHub Contents API is the correct path for single-file push. It doesn't create commits, doesn't trigger rebase conflicts, and doesn't show up in local git log — perfectly avoiding all git operation pitfalls in pipelines.

4. **reflog is the last line of defense**. After GitHub Pages 404, you can recover commits from local reflog within 90 days; beyond that you need git fsck --dangling to find dangling objects.

5. **None of the 3 defense layers can be omitted**. Prompt ban alone doesn't stop AI occasionally running git commands; 4AM check alone doesn't catch 12PM→18PM short-window incidents; .pushed_manifest alone can't survive manifest corruption.

Deployment Checklist: 5 Steps to Permanent Solidification

Solidify this incident's fixes as permanent rules (mandatory for every new workspace):

• git pull / git pull --rebase / git pull --no-rebase / git fetch origin
• git push / git push --force / git push --force-with-lease
• git rebase / git reset --hard

• Read-only: git status / git log / git diff / git ls-files
• Push single file: python3 publish-via-api.py

# Step 1: Add git operation ban snippet to all 6AM/12PM/18PM/22PM cron prompt headers
cat > /tmp/git-ban-snippet.txt << 'EOF'
⚠️ BANNED git commands (in pipeline, 4AM health-check only):

✅ ONLY allowed:
EOF

# Step 2: Add remote vs local file count comparison to 4am-health-check.py
# (script already includes main() Step 3 check_remote_vs_local())

# Step 3: Create .pushed_manifest.json template
cat > .pushed_manifest.json << 'EOF'
{"schema_version": 1, "pushed": []}
EOF

# Step 4: Add .pushed_manifest.json verification step to 4AM health check

# Step 5: Add sync_git_refs() step to 4am-health-check.py (30s timeout, non-blocking)

Applicable Scenarios

• Any engineer using cron + AI Agent to auto-maintain GitHub Pages / GitLab Pages / Vercel sites
• Any team that lets `git push --force-with-lease` enter CI/CD
• Any programmer who wants to learn disaster recovery architecture from a "git rebase disaster"

Summary and Next Steps

This incident led me to establish the permanent rule: "All AI automation pipeline git operations are delegated to a single tool, publish-via-api.py". Local main pointer never needs syncing, because GitHub Contents API push only moves files, not commits.

Next I'll write a follow-up "publish-via-api.py Push Blind Spot Practical Postmortem 2026" (already published 2026-07-30), covering 5 silent failure points + 5 step root-fix, forming a complete chain with this article: "incident root cause + push tool deepening".

(**Fact check**: All timestamps, commit hashes, commands, and filenames come from corrections.md 6/04 20:30 entry; reflog rescue commands, 4am-health-check.py sync steps, and .pushed_manifest.json format all come from Permanent Rule #24 documentation.)

👉 Join MiniMax Token Plan: AI coding acceleration for businesses

👉 Join Xiaomi MiMo Platform: API credits + 10% off first order

👉 Join Aliyun AI: Top AI products with exclusive coupons for business innovation

📌 This article was AI-assisted generated and human-reviewed | TechPassive — An AI-driven content testing site focused on real tool reviews