LangGraph Crash Recovery: What durability Modes Lose

Your worker gets killed by a deployment script while node three is halfway through charging a card. You restart it, the thread_id still resolves, the state reads back fine, and everything looks healthy — until reconciliation shows the same charge twice.

That is not a framework bug. It is the edge of the persistence contract: a checkpoint stores state, not side effects. The durability parameter decides how much of that state survives the moment the process dies. Its only official description is three lines in an API docstring, with no dedicated documentation page, so I tested it: 24 hard kills, spread across three modes and two graph shapes.

How the numbers were produced. Windows 11, CPython 3.13.12, langgraph 1.2.18 (released 2026-10-26), langgraph-checkpoint-sqlite 4.3.1, all as of publication. The kill is a real TerminateProcess from a parent process — the SIGKILL equivalent, with no cleanup handler allowed to run. Every recovery happens in a brand new process, so no in-memory object can quietly help. No affiliate links in this article.

What the three modes actually promise

langgraph/types.py line 98 defines them as three literals:

Durability = Literal["sync", "async", "exit"]

The Pregel.invoke() docstring says, verbatim:

• "sync": Changes are persisted synchronously before the next step starts.
• "async": Changes are persisted asynchronously while the next step executes.
• "exit": Changes are persisted only when the graph exits.

The default is "async", from langgraph/pregel/main.py line 2696: durability = config.get(CONF, {}).get(CONFIG_KEY_DURABILITY, "async").

All three sentences answer "when is it written". None answers "what do I lose when it crashes". The second question is the one you actually choose on.

How I tested it

Each node does two things: appends a start line and an end line to a JSONL file (each line fsync-ed, so buffering cannot fake a result), then returns a state delta.

• Linear graph: a → b → c → d, killed halfway through c, after it logged start but before it logged end.
• Parallel graph: start → p1 / p2 / p3 → join, killed halfway through p3, while p1 and p2 in the same superstep may already have finished.
• Slow backend: a SqliteSaver.put() wrapped in a 250 ms delay, imitating a networked checkpointer (Postgres or a hosted service). Without it, sync and async are indistinguishable.

Every configuration ran four times, so I report distributions instead of one lucky sample. Baseline side-effect lines for a clean run: 8 for the linear graph, 10 for the parallel one.

Linear graph: exit has no recovery path at all

With exit, the checkpoint table was empty in all four runs. graph.invoke(None, config) then raises EmptyInputError: Received no input for __start__, because in LangGraph "no input" means "continue from the stored checkpoint" — and there is none to continue from. The five side effects that already happened (a and b finished, c logged its start) are orphans.

Both async and sync kept four checkpoints and recovered, ending at nine side-effect lines: one more than a clean run. The extra line is c's second start. That is the core of the recovery semantics: the run resumes at the last completed superstep, and the interrupted superstep re-executes as a whole. The node has no idea it already called a payment API, and no mechanism tells it.

Parallel graph: the gap shows up here

On local SQLite the two modes tie at three checkpoints. Once put() costs 250 ms, async drops to a single checkpoint and repeats 6 to 7 side effects — the entire first and second supersteps run again. Sync still holds three checkpoints and repeats between zero and three.

The reason is in main.py line 3078: only durability_ == "sync" executes loop._put_checkpoint_fut.result() and blocks until the write lands. async hands the write to a background thread and keeps walking. When the backend is fast enough, the background write finishes before your kill arrives, so both modes look identical. When the backend is slow, async's in-flight window becomes a real data gap.

I measured the cost too. A clean run of the 5-node parallel graph took 0.56 / 0.55 / 0.58 seconds on a local backend, with 1 / 5 / 5 checkpoints; on the 250 ms backend it took 0.81 / 1.33 / 1.90 seconds. Sync pays exactly those extra puts — 1.1 seconds on top, or 58% of the 1.9-second total.

Why the repeat count is a range

Same graph, same kill point, repeat count drifting between 1 and 4. The reason is that the checkpoint database is really two tables: checkpoints holds a state snapshot per step, writes holds the finished deltas per task. At the moment of the kill those two are at different points. I counted writes rows at kill time: 11 for the sync runs and 14 for the async runs, on the same graph with the same four samples. Tasks whose writes already landed are reused on resume and never re-execute; the ones still in memory restart with the interrupted superstep. So "how many times it repeats" is not a constant — it depends on where the background writer happened to be.

That uncertainty is exactly why you cannot design on "the odds are low". You control two things: make the rerun safe (idempotency keys), or make the rerun smaller (push side effects to the last step). Betting on it not happening moves reconciliation into luck.

At-least-once, not exactly-once

Three conclusions you can apply today:

1. Do not design around "the graph will not rerun". Recovery is per superstep, not per node, and certainly not "resume at the line where it stopped". Anything with an external consequence — charging, emailing, opening a ticket — needs its own idempotency key, enforced downstream.

2. Push side effects to the last node, or use an outbox. Write to a local table first and let a separate consumer deliver. That moves the non-replayable part outside the graph's replay range.

3. Choose the mode by consequence, not by latency. Default async if a rerun is tolerable; sync if duplicate effects are unacceptable and the write backend is not on the same disk; exit only for one-shot batch jobs where starting over is genuinely fine.

One more comparison worth keeping in mind: an in-memory checkpointer loses the checkpoints themselves when the process restarts. The official persistence page puts it plainly — "When the process restarts, all checkpoints are lost". Good for development, wrong for anything that must continue after a crash.

Troubleshooting: five real errors

Error one — EmptyInputError: Received no input for __start__. Two triggers: recovering an exit-mode run whose checkpoint table is empty, and passing None to a thread_id that never existed. Call get_state() first and branch: create a new thread, or continue an existing one.

Error two — AttributeError: 'SyncPregelLoop' object has no attribute '_put_checkpoint_fut'. Did you mean: '_put_checkpoint'? Compile without a checkpointer, then pass durability="sync". Line 2894 warns that durability has no effect without a checkpointer, then line 3079 goes ahead and waits on a future that only exists when a checkpointer assigned it in _loop.py. It crashed 4 out of 4 times. The warning claims your setting is ignored; in reality it is fatal.

Error three — ValueError: Cannot use both checkpoint_during and durability parameters. Please use durability instead. checkpoint_during is the legacy boolean and is deprecated. Passing both during a migration triggers this immediately; drop the old flag.

Error four — a typo never raises. Write durability="syncc" and invoke returns n=5 with zero warnings. Durability is only a typing.Literal, validated nowhere at runtime, so the typo quietly falls back to the non-sync path. You find out on the next crash, not now.

Error five — sqlite3.OperationalError: database is locked. Two processes writing the same SQLite file: the first holds a write transaction, the second opens with timeout=0.3 and fails. Very common in crash-recovery architectures, where the old process is not fully dead when the new one starts. Confirm the old process exited, or move to a backend that accepts multiple writers. I took the same file apart in an eleven-round SQLite hot backup test, where a copy that opened cleanly was missing 14,500 rows.

What this test does not cover

Only SqliteSaver, single machine, synchronous invoke, no subgraphs. I did not measure the real latency distribution of Postgres or Redis (a fixed 250 ms stands in for it), did not test the astream path, and did not test how durability interacts with retry_policy or interrupt. Interruption has different semantics — it waits for a human, while a crash does not; I took that path apart in the interrupt and resume post and the streaming modes comparison. The storage layer itself, including what a checkpointer keeps and how checkpoint_ns splits, is covered in an earlier post on checkpointer internals.

If your workload is "one process, one batch, rerun the whole thing on failure", exit is fine. The moment you have long sessions, external side effects, and a k8s SIGTERM that can arrive at any second, design for at-least-once instead of designing for "it probably will not rerun".

👉 Join Xiaomi MiMo Platform: Leading AI model platform with cost-effective inference

👉 Join Aliyun AI: Top AI products with exclusive coupons for business innovation

📌 This article was AI-assisted generated and human-reviewed | TechPassive — An AI-driven content testing site focused on real tool reviews