← Back to Home

Agent Framework checkpoints: 8 ways resume fails quietly

Agent Frameworkcheckpointstate recovery

You add checkpointing to a workflow, it runs, the output is right, and the log even prints a "Created checkpoint" line. Then you ship it. That is the most common misreading of this layer: a checkpoint that writes and a resume that works are two different things, separated by half a dozen steps that never raise. I actually installed Microsoft Agent Framework 1.25.0 (published on PyPI 2026-11-07) into an isolated directory on Windows 11, installed 1.26.0 (the newest release on PyPI as of this writing, 2026-11-12) into a second one, and ran the same three-executor chains against both. Every number below comes from those runs; where I did not test something, I say so instead of guessing.

The short version: cross-process resume does work, so the layer is real. But there are far more ways to break a resume than ways to get it right, and most of the breakage does not announce itself. This article contains no affiliate links; every version number and error string below can be checked verbatim against the official package and the official documentation.

// FLOW_01Checkpoint write and resume path (measured in this article)6 NODES01STEP_01build (start_executor + checkpoint_storage)02STEP_02entry checkpoint iteration 003STEP_03run superstep 104STEP_04checkpoint iteration 105STEP_05run superstep 206STEP_06checkpoint iteration 2END_OF_FLOW
Checkpoint write and resume path (measured in this article) · diagram drawn from this article
Python
Technologies referenced · icons from Simple Icons (CC0)

Pin the version and the environment first

The environment is Windows 11 with CPython 3.13.12. Agent Framework 1.25.0 and 1.26.0 live in two separate directories (1.26.0 installed with no-deps for the core package, borrowing the remaining dependencies from the 1.25.0 directory), switched with PYTHONPATH. Every workflow is built with WorkflowBuilder and plain executors, with no model attached, so each number here is reproducible on a machine without an API key.

One boundary worth stating up front: the documentation page on checkpoints presents C#, Python and Go side by side, and the Python entry point is WorkflowBuilder(start_executor=..., checkpoint_storage=...). That entry point works as documented. But an earlier piece (the six silent traps in the wiring layer) already established that some methods listed on that same documentation site do not exist in 1.25.0, so wherever the docs and the package disagree, this article follows the package.

Checkpoint count equals superstep count plus one

A single-node workflow that runs once leaves two checkpoints (iteration 0 and 1). A two-node chain leaves three (0, 1, 2). A three-node chain leaves four (0, 1, 2, 3). The rule is stable: iteration runs from 0 to the superstep count, with one extra boundary at each end.

Iteration 0 is not "the first node finished". It is the entry checkpoint: its messages key is internal: plus the first executor id (I saw internal:step_a), it holds the input message for the run, its previous_checkpoint_id is None, and its iteration count is 0. The final checkpoint has an empty messages dictionary, because nothing is waiting to be delivered any more, only executor state.

One more detail worth recording: state has exactly four keys, _workflow_run_kwargs, _resolved_workflow_run_kwargs, _routed_workflow_run_kwargs and _executor_state. The last one maps each executor id to its saved state, and it is an empty dictionary for any executor that does not implement on_checkpoint_save.

A resume boundary sits after superstep N, not before node N

This decides whether a resume replays work or continues it. For the chain a1 to b1 to c1, a full run calls a1, b1, c1. Resuming from iteration 1 calls b1 and c1 and leaves a1 alone. Resuming from iteration 2 calls only c1. Resuming from iteration 0 reruns everything.

In other words, run(checkpoint_id=X) means "start again from whatever comes after X", and the messages already stored inside X are delivered as the next batch of input. All three resumes above produced the identical final output, so resume correctness is verifiable, but only if you treat "what runs after the resume point" as an explicit contract rather than a hope.

Resuming from the last checkpoint runs nothing and says nothing

This is the trap you will hit on the happy path. In the same three-node chain, passing iteration 3 (the checkpoint written after the run finished) makes run() return normally with an empty output list and zero executor calls. No exception.

From the caller's side it looks exactly like a successful resume with no new output. What it actually means is that every superstep was already complete when you resumed. Put that in a retry helper with a rule like "resume from the latest checkpoint if one exists" and a finished job gets classified as "resumed, empty result", after which the empty result is written to the database as the real result.

There is only one defence: compare the target checkpoint's iteration against the number of supersteps in the graph, or resume only from a checkpoint you know is not the last one. get_latest() returns precisely the last checkpoint, which makes it the single worst candidate for a resume point.

A failed checkpoint write does not fail the workflow

This is the most expensive finding in the article, and the reason it should not live only in the log. Give an executor an on_checkpoint_save that returns an instance of a custom class (mine was Order(oid, qty)) and run it: run() returns the expected output, everything looks successful, and the checkpoint directory is empty. Not fewer files, zero files.

The log line reads:

WARNING | agent_framework._workflows._runner | Failed to create checkpoint at iteration 0: Failed to decode pickled checkpoint data: Checkpoint deserialization blocked for type 'ckptmodel:Order'. ... Note that this does not fail the workflow run. The next successfully-created checkpoint will be parented to the last successful checkpoint

Two things to notice: the level is WARNING, not ERROR, on the agent_framework._workflows._runner logger, and it explicitly states that it does not fail the workflow run. The implementation agrees, since the whole save step in _runner.py sits inside except Exception and only logs a warning before continuing. Raise your logging threshold on the agent_framework logger to ERROR and the failure is invisible while the workflow still reports success.

The cause is on the decode side. The FileCheckpointStorage docstring says deserialization is restricted by default to built-in safe types (primitives, datetime, uuid and so on), all internal agent_framework and agent_framework_orchestrations types, and the OpenAI SDK's openai.types. The file on disk is JSON, but complex objects are embedded as __pickled__ plus base64 (I opened one: eleven top-level keys, and messages carries exactly that shape), so reading them goes through a restricted unpickler. There are two ways to allow an application class: pass allowed_checkpoint_types=["module:ClassName"] to the storage constructor, or call agent_framework.register_checkpoint_type(cls) to register it globally.

The measured contrast is clean. With a default storage the same executor produced one checkpoint (only the entry checkpoint, whose state held an empty list at that moment). With allowed_checkpoint_types=['ckptmodel:Order'] it produced two, and load() returned Order(X1,1) inside _executor_state.

A misspelled allowlist entry fails silently too

The counter-example: pass allowed_checkpoint_types=['Order'], with no module prefix. Constructing the storage raises nothing and warns about nothing, and the run behaves exactly as if the argument were absent, one checkpoint and a lost custom state.

So the parameter is of the quietly-ignored kind: right format or nothing. Same behaviour on 1.26.0. The only reliable check is not "did I set the parameter" but "how many checkpoint ids exist afterwards", and by the rule above a two-node chain should report three. Anything less means a checkpoint never landed.

The graph hash sees structure, not code

Every checkpoint carries a graph_signature_hash. Changing the graph has a clear outcome: pointing a1 at c1 instead of b1 makes the resume throw

WorkflowCheckpointException: Workflow graph has changed since the checkpoint was created. Please rebuild the original workflow before resuming.

Changing an executor id (from x to y) or a class name triggers the same exception. Renaming the workflow from s8 to renamed does not, and the resume still works.

The interesting case is the third one. Changing only the body of the executor leaves the hash identical. I changed a worker from appending |v1 to appending |v2-changed-body, and both builds produced the same hash (starting 324fa621effcaf94). The resume was accepted without complaint, the message handed to the downstream node was the one produced by the old code, and the output stayed tail:x|v1.

Reading _compute_graph_signature in _workflow.py explains it: the signature collects start_executor, each executor's module.ClassName, and each edge group's source, target and condition, sorted and hashed with sha256. Function bodies, default arguments and business logic are not in it. So "I changed the code but not the shape" produces a resume that silently mixes new code with old data, which is exactly the scenario where a data mismatch is hardest to spot. The only defence is your own version marker: put a logic version in the workflow name or in the checkpoint directory path, and change the directory when the logic changes.

list_checkpoints is not ordered by superstep

For the three-node chain, list_checkpoints(workflow_name=...) returned iteration sequences of [1, 2, 0] in one run and [2, 1, 0] in another. Never [0, 1, 2]. The implementation explains it: the file backend iterates for file_path in self.storage_path.glob("*.json") with no sorting at all.

That directly undercuts the documented sample, which says "assume we want to resume from the 6th checkpoint" and then indexes checkpoints[5]. On the file backend, position six and superstep six are unrelated. If you want superstep order, sort by iteration_count yourself. get_latest() does hold up: it picks the newest timestamp and resolves ties through the lineage chain via previous_checkpoint_id, and it returned iteration 2 rather than a lucky guess.

The same implementation carries a second edge: listing decodes every file and only logs a warning on failure (Failed to read checkpoint file ...), so a checkpoint that cannot be decoded silently disappears from the list while load(id) on that same id raises properly. "Not in the list" does not mean "does not exist".

Every checkpoint is a full snapshot, and nothing prunes it

I deleted the iteration 1 checkpoint and then resumed from iteration 2. The result was correct and only c1 ran. I also deleted the entry checkpoint and ran a fresh, unrelated run, which completed normally. Checkpoints hold complete state rather than deltas, so a resume does not need its ancestry.

The good news is that old checkpoints can be deleted. The bad news is that nothing deletes them for you. The CheckpointStorage interface has exactly six methods, save, load, get_latest, list_checkpoints, list_checkpoint_ids and delete, with no TTL, no retention count and no sweep parameter, and the documentation page does not discuss retention either. The measured cost: one two-node run leaves three checkpoints, and two resumes take the directory to six (+100%). With a 20 KB message payload and 20 runs, the directory held 60 files and about 1,097 KB, and a single list_checkpoints() call took roughly 18 ms because it fully decodes every file. list_checkpoint_ids() delegates to it and took 19 ms, so the two cost the same. Storage grows linearly and so does listing.

The timestamp is UTC while the file time is local

The stored timestamp carries an offset, and I measured 2026-11-14T22:12:49.750179+00:00 for a file whose mtime read 2026-11-15T06:12:49 in local time on a +0800 machine. Eight hours apart, same instant.

Compare "checkpoint time" against a local business timestamp, for example "only resume checkpoints created today", and those eight hours will classify any job that crossed midnight as yesterday's work. Convert one side, or use mtime for both, but do not mix them.

Cross-process resume works if the graph rebuilds to the same fingerprint

This is the promise the layer has to keep, so I used two processes. Process A (PID 20032) wrote the checkpoints and printed the id of iteration 1. Process B (PID 16244) resumed from that id, read a graph_signature_hash identical to A's (starting a48f53cff8c2b80a), ran only stage2, and produced the same output as A's full run.

So the fingerprint is stable across processes and contains nothing like a memory address. As long as executor ids, class names, module paths and edge structure are unchanged, another process rebuilds the same fingerprint. The corollary is that a three-part refactor invalidates every historical checkpoint at once: rename the class, move the module (say from app.nodes to app.workflow.nodes), or change an executor id. The C# documentation is explicit that agent ids must be stable; the Python implementation follows the same rule, except the exception only says the graph changed and never says where.

How this layer divides work with LangGraph

The LangGraph checkpointer piece covers what changes once persistence is wired up. This article covers what a checkpoint file actually records and which step a resume starts from. Do not merge the two in your head: the LangGraph side is driven by a thread concept, while this layer has no thread at all and the caller must pass checkpoint_id into run() explicitly, with no default to fall back on.

Nor does this help with the waiting-heavy concurrency work from the thread pool comparison, because checkpoints do not touch scheduling, only the question of where to continue. It earns its cost when a single superstep is expensive (a model call, an external write) and a rerun has side effects. For the crash-side question of what each durability mode actually loses, the durability piece walks through it mode by mode.

Troubleshooting: four real errors and what to do

Error 1: WorkflowCheckpointException: Workflow graph has changed since the checkpoint was created. Please rebuild the original workflow before resuming.

Triggered by changing an edge, an executor id or an executor class name, since the module path is part of the signature. Either rebuild the exact structure or abandon the old checkpoints. It never tells you which node differs, so compare those three things yourself.

Error 2: WorkflowCheckpointException: No checkpoint found with ID not-a-real-id

Triggered by passing an id that does not exist or was deleted. Check with list_checkpoint_ids(workflow_name=...) first. Note that passing the wrong workflow name does not raise this; it returns an empty list, which reads as "there were never any checkpoints".

Error 3: TypeError: FileCheckpointStorage.list_checkpoint_ids() missing 1 required keyword-only argument: 'workflow_name'

All three query methods take workflow_name as keyword-only. Passing it positionally or omitting it raises TypeError immediately. Write list_checkpoints(workflow_name=wf.name). The same TypeError appears on list_checkpoints() and get_latest(), with only the method name differing.

Error 4: Checkpoint deserialization blocked for type 'ckptmodel:Order'. To allow this type, either include its 'module:qualname' key in the 'allowed_types' set passed to 'decode_checkpoint_value', or add it to 'allowed_checkpoint_types' on your checkpoint storage (for example, 'FileCheckpointStorage.allowed_checkpoint_types').

Triggered by any custom type in executor state that has not been allowed. It is a WARNING, not an exception: the workflow succeeds and the checkpoint is silently missing. Fix it with the module:qualname form, then re-check against the hard rule that the checkpoint count equals the superstep count plus one.

FAQ

Q: Can I read or edit a checkpoint file directly?

Reading works, since the top level is JSON with eleven keys (workflow_name, graph_signature_hash, checkpoint_id, previous_checkpoint_id, timestamp, messages, state, pending_request_info_events, iteration_count, metadata, version). Editing breaks it, because complex objects are __pickled__ blobs in base64. The security point matters more: the documentation states that checkpoint storage is a trust boundary and that you must never load checkpoints from untrusted or tamperable sources. The restricted unpickler blocks types outside the allowlist, not files that were swapped.

Q: Can InMemory and File backends be mixed?

They share the CheckpointStorage protocol, but behaviour differs: InMemory lists in insertion order while File lists in filesystem order. I ran two differently named workflows against one InMemory storage and each kept its own three checkpoints without bleeding into the other, with the same keyword-only workflow_name requirement. Cross-process resume needs File or an external store such as Cosmos, because InMemory dies with the process.

Q: Does a resume re-run nodes that already succeeded?

It depends on the checkpoint. Resuming from iteration 1 skips everything before superstep 1, while resuming from the entry checkpoint replays the whole run. Resume is not an idempotency guarantee; that has to live in your executors. The fact that deleting a parent checkpoint does not break a child proves checkpoints are full snapshots, which is a different property from re-execution: the resume skips completed supersteps rather than deduplicating work.

Q: Did any of this change in 1.26.0?

I re-ran every experiment. The checkpoint count rule, the no-op resume from the last checkpoint, the silent loss after a blocked type (still zero files), the code-blind graph hash, the unsorted listing (this time [2, 1, 0]), and the idempotent delete (True then False) all matched. The same graph also hashes identically on both versions. So reading these conclusions as "true on 1.25.0 and 1.26.0" is safe; I did not verify anything after that.

When checkpointing is worth turning on

Three conditions, and all three have to hold: a single superstep is expensive enough to justify the storage cost, a rerun has external side effects, and you accept that the caller names the resume point. Three situations to avoid: jobs that finish in seconds, executors holding many custom objects with no allowlist configured (you will get a pile of invisible failures), and using get_latest() as the resume point, which is the last checkpoint and therefore a no-op.

The personality of this layer, in one line: it hands you the decision about where to continue, and when you get that decision wrong it prefers to do nothing quietly. The earlier wiring-layer piece found six silent traps between nodes; this is the same family of problem one layer down at storage. What they share is not a hard API but a default failure mode that looks like success.

👉 Join Xiaomi MiMo Platform: Leading AI model platform with cost-effective inference

👉 Join Aliyun AI: Top AI products with exclusive coupons for business innovation

📌 This article was AI-assisted generated and human-reviewed | TechPassive — An AI-driven content testing site focused on real tool reviews

🔗 Recommended Tools

These are carefully selected tools. Using our affiliate links supports us to keep producing quality content:

As an Amazon Associate, I earn from qualifying purchases at no extra cost to you.

☁️ DigitalOcean Cloud ⚡ Vultr VPS 🤖 QoderWork CN (Refer & Earn) ☁️ Aliyun AI Products 📚 WordPress Books 🧠 AI & LLM Books 🏗️ System Design Books 🐳 Docker & K8s Books 🐧 Linux Admin Books 🐍 Python Books 🗄️ Database Books 🚀 DevOps & SRE Books 🤖 Xiaomi MiMo Platform
← Back to Home