← Back to Home

Task retries in Celery and huey: three runs, one side effect

Celeryhueytask queue

The part of a task queue that gets misread most often is not whether it retries. It is how many times it retries, and how many times the side effect runs. The three mainstream Python queues answer those two questions with three different counting conventions, and none of them warns you when the convention is not the one you assumed.

I actually ran the same set of tasks against huey 3.5.4 and Celery 5.6.3, installed into an isolated directory on CPython 3.13.12 and Windows 11: one task that always fails, one that fails twice and then succeeds, and one submitted twice with a fixed id. Every number below came out of that run, not out of a documentation page.

TL;DR: With default settings, huey's retries=2 means three total executions, and Celery's max_retries=2 means three as well, but a retry without an explicit countdown waits 180 seconds. Neither library deduplicates on task_id. Celery's backoff carries jitter by default, so the documented "first retry after 1 second" is only a ceiling; the measured mean is half of it. acks_late does not buy you "never lost", it buys you "possibly duplicated".

Three counting conventions on one table

LibrarySettingWhat the docs sayMeasured total executions
huey 3.5.4retries=2retry twice on unhandled exception3
Celery 5.6.3max_retries=2 + autoretry_forretry at most twice3
Celery 5.6.3max_retries left at defaultclass attribute Task.max_retries = 34 (from source, not measured)
No retry configured——1

The first two rows ran three rounds each and stayed stable. The third row is a source constant; I did not run it separately, so treat it as source evidence rather than measurement.

The row that actually bites is the last one. The default is one attempt. A task with no retry configuration fails once and stays failed. The queue will not cover for you.

In huey, retries is a countdown, not a total

huey counts down. The first line of _requeue_task() in huey/api.py is task.retries -= 1, and then the task goes back on the queue. So a task declared with retries=2 fails, drops to 1, fails again, drops to 0, fails a third time, and only then does final = not task.retries become true and the error reach the result store.

That is why the table says three and not two. The documentation wording — "number of times to retry the function if an unhandled exception occurs" — is accurate, but reading it as a total is off by a factor of two.

The error shows up before the retries are finished

This is where huey generates the most false alarms. SqliteHuey defaults store_intermediate_errors to True; the docs say "Defaults to True for backwards compatibility". The meaning is that when a task fails but still has retries left, the exception is written into the result store right away.

I measured it. A task with retries=3, retry_delay=1 failed once, and 0.5 seconds later the result store already returned a TaskException wrapping ValueError('attempt-1') — while the task was still queued for another attempt and eventually succeeded.

If your alerting rule is "a TaskException in the result backend means page someone", every task that eventually succeeds will page someone first. Either set store_intermediate_errors=False, or read the retries field in the Error metadata and only alert when it has hit zero.

One more counterintuitive detail: h.result() uses preserve=False by default, so reading a result pops it out of storage. My first probe read the same result twice and got None on the second read; I nearly recorded that as "the retry ate my result". Use preserve=True if you need to read twice.

Once retries really are exhausted, the shape is TaskException: ValueError('boom-A') — identical to the intermediate state. Only the retries field tells them apart.

Celery's 180 seconds: you think it is retrying, it is queueing

Task.default_retry_delay is 180 as a class attribute, and Task.retry() contains:

if not eta and countdown is None:
    countdown = self.default_retry_delay

So a bare raise self.retry() waits three minutes before the task re-enters the queue. That is not a bug; it is a default written for a time when the main fear was hammering an overloaded downstream service. But in the modern mental model, "retry" means "try again in a moment".

If you want second-level retries you have to pass countdown=1, or use autoretry_for together with retry_backoff.

A direct call is not a retry

Task.retry() opens with:

if request.called_directly:
    raise_with_context(exc or Retry('Task can be retried', None))

In my run, calling manual("direct") directly raised ValueError, not Retry, with exactly one execution. Call a task function by hand in a REPL and you see the original exception; the retry path never runs.

The comment above that block reads "Not in worker or emulated by (apply/always_eager), so just raise the original exception". The comment and the behavior disagree: apply() takes the is_eager branch and, in my run, did retry three times. The comment lumps apply in with direct calls, but only called_directly actually skips retries.

Eager mode swallows the backoff

This is the classic trap when you validate retry pacing in a unit test. I ran a task with retry_backoff=True, max_retries=2 under task_always_eager=True:

Three runs in ten milliseconds. The countdown is computed and stored in the Retry signature, but apply() catches the Retry and re-executes in place, with no sleep at all. The same configuration on a real worker waits 1 second and then 2 seconds; under eager it waits 0 and 0.

Eager mode can prove how many times a task retried and which exception finally surfaced. It cannot prove anything about pacing. For pacing you need a real worker, or a direct unit test of get_exponential_backoff_interval().

What gets raised at exhaustion depends on `exc`

Same setup, two variants:

Both ran three times. The documentation is accurate here: the original exception is re-raised when exc was supplied, and MaxRetriesExceededError is raised when it was not.

This matters for your except clauses. If a fallback handler catches MaxRetriesExceededError and you happen to pass exc, that handler never runs.

Jitter is on by default, so "1 second" is a ceiling

The docs say retry_jitter is set to True by default, and separately that with retry_backoff=True "the first retry will have a delay of 1 second". Both statements are true, and together they read like "the first retry always waits one second".

With full_jitter=True the delay is random(0, ceiling). I sampled get_exponential_backoff_interval() 2000 times per row:

Retry numberSampled minSampled maxSampled meanDocumented ceiling
1010.51
2021.02
3042.04
4084.08

The mean is exactly half the ceiling and the minimum is 0. In other words, the expected delay you actually get is only 50% of the number the docs put in front of you. Exponential backoff in the default configuration is "exponential ceiling plus uniform noise", not "exponential wait".

There is a configuration-layer detail worth recording: DEFAULTS['task_retry_jitter'] is None, not True. What makes the effective default True is this line in celery/app/autoretry.py: getattr(task, 'retry_jitter', True). Reading the config table, seeing None, and concluding jitter is off is reading the wrong layer.

`task_id` is an identity, not a lock

I assumed a fixed task_id would work as an idempotency key. Measured:

Neither library deduplicates. task_id identifies a message so you can look up its result, revoke it, or trace it. It is not a promise that the id runs once. Anyone relying on it for deduplication finds out on the first duplicate consumption, when it blocks nothing.

`acks_late` does not buy "never lost", it buys "possibly duplicated"

The Celery docs are unusually blunt:

the default behavior is to acknowledge the message in advance, just before it's executed, so that a task invocation that already started is never executed again.

The ack happens before execution, so a worker killed mid-task loses the message; nothing is redelivered. Turning on acks_late moves the ack to after execution:

This means the task may be executed multiple times should the worker crash in the middle of execution. Make sure your tasks are idempotent.

The FAQ on the same site adds the line that matters most:

if the task raises an exception it's still acknowledged!

An exception still acks the message. Retries in Celery are therefore not driven by the ack mechanism at all; they are driven by retry(). acks_late governs "the process died", not "the code raised".

Three source constants in Celery 5.6.3: task_acks_late = False, task_reject_on_worker_lost = None, worker_prefetch_multiplier = 4. The second one is None, not False, which means "follow the semantics of acks_late", not "disabled".

Evidence boundary: this machine has no Redis or RabbitMQ, so I did eager-mode measurements, source-constant extraction, and direct quotation of the official docs. I did not run a real redelivery scenario. To know whether your broker redelivers after a hard kill, you have to kill a worker on your own infrastructure.

Troubleshooting: three errors I actually hit

AttributeError: 'bytes' object has no attribute 'eta'

I tried to drive the queue without a consumer: task = h.storage.dequeue() followed by h.execute(task). SqliteHuey.dequeue() returns serialized bytes, not a Task. Deserialize it first with h.deserialize_task(data), or just use h.create_consumer(), which is the production path and the one I ended up on.

ValueError: signal only works in main thread of the main interpreter

Putting consumer.run() into a threading.Thread crashed immediately. Consumer.start() calls _set_signal_handlers(), and signal.signal() can only be registered from the main thread. In tests, either stub _set_signal_handlers to a no-op inside the thread, or start the consumer in a separate process — the latter is closer to production and hides less.

ProgrammingError: Error binding parameter 2: type 'Result' is not supported

I passed a Result object to h.result(r) when it wants the task_id string. The correct form is h.result(r.id). The error comes out of sqlite3 parameter binding and gives no hint that the argument type was wrong; it took me a while to connect the dots.

How to get idempotency, four options by cost

If a task "ran twice and produced three rows", do not start by lowering the retry count. Check whether it has any of those four properties. Retry count only controls how many times it repeats; idempotency design decides what a repeat does.

Picking between the three, in one line each

The dependency surfaces and the silent-failure lists for all three are in the November 1 comparison. This piece only answers what happens after a failure, so the two do not overlap.

FAQ

What is the difference between retries=0 and retries=-1?

In huey, 0 means no retry. A negative number keeps decrementing in _requeue_task(), and final = not task.retries is never true for a negative value, which adds up to unlimited retries. If you want unlimited retries, confirm that behavior deliberately instead of guessing with -1.

What does Celery's max_retries=None mean?

"A value of None means task will retry forever." It does not mean "no retry". Use 0 for that.

Why does my task rerun instantly even with retry_backoff=True?

Check whether you are running under task_always_eager=True. Eager mode never sleeps, which is exactly why my run finished three executions in 0.010 seconds.

My task executed twice — is that acks_late?

First check whether acks_late is on at all. The default task_acks_late is False, acking before execution, so a crash normally does not redeliver. Duplicates are far more likely to come from retry() being called, or from the producer publishing twice.

Closing

The three numbers worth keeping are not about which library is better: retries=2 equals three executions, the default countdown is 180 seconds, and task_id deduplicates nothing. All three are documented, and none of them raises an error when you get it wrong.

Queue defaults are almost always chosen in favor of reliability, and the only way to implement reliability across machines is at-least-once delivery. Once delivery is at-least-once, duplication is the normal case and idempotency is not optional.

No affiliate links in this article. Every number here is either measured locally or quoted verbatim from the official documentation.

👉 Join Xiaomi MiMo Platform: Leading AI model platform with cost-effective inference

👉 Join Aliyun AI: Top AI products with exclusive coupons for business innovation

📌 This article was AI-assisted generated and human-reviewed | TechPassive — An AI-driven content testing site focused on real tool reviews

🔗 Recommended Tools

These are carefully selected tools. Using our affiliate links supports us to keep producing quality content:

As an Amazon Associate, I earn from qualifying purchases at no extra cost to you.

☁️ DigitalOcean Cloud ⚡ Vultr VPS 🤖 QoderWork CN (Refer & Earn) ☁️ Aliyun AI Products 📚 WordPress Books 🧠 AI & LLM Books 🏗️ System Design Books 🐳 Docker & K8s Books 🐧 Linux Admin Books 🐍 Python Books 🗄️ Database Books 🚀 DevOps & SRE Books 🤖 Xiaomi MiMo Platform
← Back to Home