← Back to Home

Blog Pipeline Silent Death for 20 Days: Clarity Traffic Camouflage vs git log Truth

pipeline monitoringDevOpsSREblog automationgit loghealth checklong-tail traffic

I opened the Clarity dashboard at 11PM that night, saw 32 sessions and 27 unique users, and thought "traffic is up again." Then I ran git log --since="2026-08-10" --oneline --grep="auto: publish" out of habit. It returned nothing. I rubbed my eyes, assumed I'd typed the parameters wrong, tried again — still nothing.

That's how I discovered my blog publishing pipeline had been "silently dead" for 20 days. After August 10th, not a single new article was published. But the Clarity dashboard kept reporting beautiful numbers every day, the 4AM health check kept saying "all systems normal," and the entire monitoring stack was collectively blind.

The Crime Scene: What Happened After August 10th

August 10th was the last normal publish day. Three articles went out: a USB-C global travel adapter roundup, a WordPress ActivityPub integration piece, and a post-mortem about git pull rebase causing 177 articles to 404. After that — complete silence.

But nobody, including my own automated monitoring scripts, noticed the silence. Here's why:

Clarity kept reporting traffic. On August 28th, it even reported 32 sessions (a recent high). The top pages were all old articles: the Claude Code API Key configuration piece (published May 31st) was pulling 7 sessions daily for three consecutive days. The GLM-5 local deployment test (April 27th) was getting 2 sessions daily. The Paperless-ngx self-hosted document management article (April 29th) was also getting 2 sessions daily.

The long-tail SEO effect of old articles was accumulating. Even with zero updates for 20 days, traffic kept growing. This made everyone think things were fine.

The 4AM health check ran daily but only checked two things: First, whether pushed_manifest.json was in sync (yes, it had been syncing fine). Second, whether the remote main branch and local file count differed by more than 10 files (no, because neither side had changed). It never checked the most fundamental metric: "Were any new articles published in the last 24 hours?"

False Signals: Three Illusions of "Normal"

Illusion #1: Clarity Traffic = Blog Health

This was the deadliest cognitive error. Clarity reports "how many people visited the website," not "whether the website is being updated." A blog that has accumulated 600+ articles can sustain 20-30 daily sessions on old content alone, even with zero updates — for at least 3-6 months.

I was using Clarity session count as a proxy for blog health. That's like judging "Is the company still operating?" by checking "Does the company bank account have money?" — the balance could be from last month's revenue, not evidence that this month's production line is running.

Illusion #2: pushed_manifest.json Sync = Publishing is Fine

The 4AM health check's logic was: if all files recorded in pushed_manifest.json can be found on the remote main branch, consider "push is working." But pushed_manifest.json's last update was August 10th — there were simply no new records to sync. Checking "Are previously pushed files still on the remote?" and checking "Have any new files been pushed?" are fundamentally different questions.

Illusion #3: git commits Exist = Content is Updating

The 4AM health check did run git commit daily — but those were just chore: sync pushed manifest [4am-health-check] entries, commits generated by the health check script itself, not article publications. The git log showed a commit every day, but not a single one was auto: publish.

Root Cause: Why the Entire Monitoring Stack Failed

The root cause wasn't a bug in any single script. It was that the entire monitoring stack only checked whether the pipe was clear, never whether water was actually flowing through it.

Here's a diagram of the problem:

Monitor Layer       What It Checked             Status 8/10-8/30
──────────────────────────────────────────────────────────────────
Clarity             Website traffic              ✅ 20-32 sessions/day (old articles)
4AM health-check    pushed_manifest sync         ✅ No new files to sync
4AM health-check    Remote vs local file diff    ✅ Diff <10 (neither side changed)
4AM health-check    git commit exists            ✅ Daily chore commits
──────────────────────────────────────────────────────────────────
MISSING:            auto:publish count (24h)     ❓ Never checked
MISSING:            drafts/ queue pile-up        ❓ Never checked
MISSING:            cron task last execution     ❡ Never checked

Three layers of "normal" perfectly concealed one fact: everything had completely stopped.

The Hack: 3 Lines of Shell Script to Plug the Hole

The fix doesn't require rebuilding the monitoring stack. It just needs a Step 0 added at the very beginning of the 4AM health check:

# Step 0: Check if any auto: publish commits exist in the last 24 hours
AUTO_PUBLISH_COUNT=$(git log --since="24 hours ago" --oneline --grep="auto: publish" | wc -l)
if [ "$AUTO_PUBLISH_COUNT" -eq 0 ]; then
  echo "⚠️ WARNING: No new articles published in the last 24 hours! Pipeline may be dead."
  # Hook into your alerting system (email/Telegram/Slack)
  ALERT_SENT=true
fi

The core logic is: **use git log as the single source of truth.** Don't look at Clarity (fooled by old article long-tail traffic). Don't look at pushed_manifest (only checks existing records, not new additions). Only check one thing — whether auto: publish appeared in a git commit in the last 24 hours.

If your pipeline uses a different commit message format, just swap --grep="auto: publish" for your own tag.

Advanced: Consecutive-Day Alert Escalation

The 3-line script only solves "can detect the problem." You also need to solve "the problem gets ignored":

# 3 consecutive days of 0 publishes → escalate alert
CONSECUTIVE_ZERO_DAYS=$(cat /tmp/pipeline-zero-publish-streak 2>/dev/null || echo 0)
if [ "$AUTO_PUBLISH_COUNT" -eq 0 ]; then
  CONSECUTIVE_ZERO_DAYS=$((CONSECUTIVE_ZERO_DAYS + 1))
  echo "$CONSECUTIVE_ZERO_DAYS" > /tmp/pipeline-zero-publish-streak
  if [ "$CONSECUTIVE_ZERO_DAYS" -ge 3 ]; then
    echo "🚨 CRITICAL: Pipeline has been dead for ${CONSECUTIVE_ZERO_DAYS} days! Manual intervention required."
    # Escalation: email + Telegram + disable 12PM/18PM/22PM cron jobs
  fi
else
  echo "0" > /tmp/pipeline-zero-publish-streak
fi

Muscle Memory Anti-Recurrence: My Monitoring Checklist

After this incident, I added these checks to the 4AM health check script, sorted by priority:

PriorityCheckAlert ConditionMethod
P0auto:publish count (24h)=0git log --grep
P1Consecutive zero-publish days≥3 days/tmp counter file
P2drafts/ queue file count>20 filesls + wc
P3cron task last execution>25 hours no executioncrontab log

P0 is the 3-line shell script above. P1 is the escalation version. P2 checks whether the draft queue is piling up (20+ draft files means the pipeline's consumption capacity has broken). P3 is the lowest-level check — if the cron job itself isn't running, all the above checks are meaningless.

3 Things I Learned From This Incident

First, old article long-tail traffic is both medicine and poison. It keeps your blog looking "healthy" even when you've completely stopped publishing. This is both the value of your SEO asset and the source of your monitoring blind spot. You need to monitor "content update frequency" and "traffic metrics" as separate signals — one cannot substitute for the other.

Second, "clear pipe" does not mean "water is flowing." pushed_manifest sync, remote file count consistency, git commit existence — these are all "clear pipe" checks. But you need an additional check for "how much new content was pushed in the last 24 hours" — the "water flow" metric.

**Third, git log is the best audit trail for automated pipelines.** It won't be fooled by Clarity's long-tail traffic, won't be confused by pushed_manifest sync status, won't be deceived by chore commit illusions. As long as your pipeline tags its commit messages (like auto: publish), git log is the only trustworthy source for answering "Has any new content been published?"

---



👉 Join MiniMax Token Plan: AI coding acceleration for businesses

👉 Join Zhipu Coding Plan: GLM-4.6/GLM-5 coding packages, China-stable, pay-per-token unlimited

👉 Join Aliyun AI: Top AI products with exclusive coupons for business innovation

📌 This article was AI-assisted generated and human-reviewed | TechPassive — An AI-driven content testing site focused on real tool reviews

🔗 Recommended Tools

These are carefully selected tools. Using our affiliate links supports us to keep producing quality content:

☁️ DigitalOcean Cloud ⚡ Vultr VPS ⭐ MiniMax Token Plan 🧩 Zhipu Coding Plan 🎁 Zhipu 20M Tokens Gift 🤖 QoderWork CN (Refer & Earn) ☁️ Aliyun AI Products 📚 WordPress Books 🔍 WordPress SEO Books 🌐 Web Hosting Books 🐳 Docker Books 🐧 Linux Books 🐍 Python Books 💰 Affiliate Marketing 💵 Passive Income Books 🖥️ Server Books ☁️ Cloud Computing Books 🚀 DevOps Books
← Back to Home