huey vs APScheduler vs Celery: Seven Silent Failures
Running a scheduled job on a single machine is rarely hard. Working out whether it ran is the hard part. I tested huey 3.5.3, APScheduler 3.11.3 and Celery 5.6.3 on Windows 11 with CPython 3.13.14, trying to answer one question: why does the same requirement fail in three different ways, quietly, in three libraries?
The short answer is uncomfortable. None of the dangerous parts are on the first page of the docs. They are in the defaults. Defaults never raise; they just make your job run fewer times than you think.
TL;DR: When a job "did not run but did not error", check four things first: whether a huey consumer is actually running, whether APScheduler's misfire_grace_time is still the default 1 second, whether your Celery broker is memory://, and whether the main thread exited before the job fired. Every number below comes from a run on my machine. No affiliate links in this article.
The three libraries on one table
Most comparison posts start with throughput. On a single machine, throughput is rarely the bottleneck; operability is. The real differences look like this:
| huey 3.5.3 | APScheduler 3.11.3 | Celery 5.6.3 | |
|---|---|---|---|
| External service needed | No (SqliteHuey uses stdlib sqlite3) | No | Yes, a broker |
| Process model | Separate consumer process | Background thread in-process | Separate worker process |
| Persistence by default | Yes (SQLite / Redis / PostgreSQL) | No (memory jobstore) | Configure a result backend |
| Built-in retries | Yes (retries, retry_delay) | No, write your own | Yes (autoretry_for) |
| Scheduling syntax | crontab(minute='*/5') | cron / interval / date triggers | beat + crontab |
| Works with no external service | Yes | Yes | No (memory:// is threads only) |
That last row is where this article starts. On a machine without a broker, you cannot observe Celery's real behaviour. You observe a stand-in, and the stand-in gives you false confidence.
huey: three silent failures
huey is deliberately plain. A task is serialized, pushed into a queue store, and another process pops it. Few dependencies, but there is a person-shaped gap between "something is in the queue" and "something happened".
Failure 1: the task is queued and nobody executes it
Calling a @huey.task() function does not run it; it enqueues it. With no consumer running, the queue grows and the Result handle returns None forever:
from huey import SqliteHuey
h = SqliteHuey('q1', filename='q1.db')
@h.task()
def add(a, b):
print('executed')
return a + b
r = add(1, 2)
print(r()) # None
print(h.storage.queue_size()) # 1
On my machine: r() returned None, queue_size() was 1, the side effect never happened, and there was no exception and no log line. After I manually ran h.execute(h.dequeue()), r() returned 3.
The trap is semantic. None means "not executed yet", not "execution failed". A failed task makes Result.get() raise TaskException. If your health check is if result is None, you are reading "not ready" as "broken".
Failure 2: the result is destroyed on first read
I hit this while writing assertions: assert once, print once, and the value is gone. The cause is in Result._get(), which calls huey.get_raw(task_id, peek=False), and peek=False routes to storage.pop_data(): reading deletes.
The official API docs say it plainly:
> preserve (*bool*) – when set to True, this parameter ensures that the task result will be preserved after having been successfully retrieved. Ordinarily, Huey will discard results after they have been read, to prevent the result store from growing without bounds.
Measured:
r = mul(6, 7)
h.execute(h.dequeue())
print(r()) # 42
print(h.result(r.id)) # None - the store is already empty
print(r.get(preserve=True)) # pass this when you need to re-read
The same Result object still returns 42 on a second call because it caches _result. A fresh Result does not. So "it works locally but the result endpoint returns nothing" is usually about how you read, not where you store.
Failure 3: an unregistered task name deletes the message
Rolling deploys leave two code versions alive at once. If a message carries an old task name and the consumer does not register it, huey does not skip it or retry it. It raises during dequeue, after the message is already gone:
huey.exceptions.HueyException: __main__.legacy_name not found in TaskRegistry
The measurement worth remembering: queue_size() was 1 before the exception and 0 after. Huey.dequeue() pops the message from storage first, then calls deserialize_task(), and TaskRegistry.string_to_task() raises when the name is unknown. There is no rollback path.
The consumer loop does not save you either:
try:
task = self.huey.dequeue()
except Exception:
self._logger.exception('Error reading from queue')
self.sleep()
One traceback, then backoff. The task never returns. Treat the task name as a persistence contract: rename with an alias, never in place.
huey's SQLite store: one timeout parameter, two meanings
SqliteHuey takes a timeout parameter. The docs are thin; the source comment reads # Busy timeout in seconds, default is 5. The problem is that it also sets a second deadline.
huey 3.5.2's changelog has this line:
Fixed a "database is locked" problem when multiple processes open a new sqlite database at the same time. See #925.
The fix lives in storage.py:
def _set_journal_mode(self, conn):
# Switching a new database to WAL upgrades a read lock to a write
# lock, which can fail under concurrency.
deadline = time.monotonic() + self._timeout
while True:
try:
conn.execute('pragma journal_mode="%s"' % self._journal_mode)
return
except self.operational_error as exc:
if 'database is locked' not in str(exc):
raise
elif time.monotonic() >= deadline:
raise
time.sleep(0.01)
So timeout is both the SQLite busy timeout and the retry window for creating the database. Shrinking it (a common "performance" suggestion) closes that window. Three processes, 400 tasks each:
timeout | Tasks stored | Process outcome |
|---|---|---|
0 | 400 (800 lost, 66.7%) | 2 processes rc=1, sqlite3.OperationalError: database is locked |
5 (default) | 1200 | all 3 processes rc=0 |
The two failing processes exited. They did not retry into success. If systemd or nohup starts several enqueue processes, one crash means one fewer writer while the queue still grows, which reads as healthy.
One boundary: WAL allows readers alongside a single writer, and huey sets isolation_level = None (autocommit), so multi-process writes work. Long transactions do not. Same mechanism as WAL hot backups, which is covered in SQLite hot backup: 14,500 rows missing.
APScheduler: three defaults, zero errors
APScheduler is subtler because there is no queue at all. Jobs run in an in-process thread pool, and both failures and skips only reach the log. The three defaults are printable:
python -c "from apscheduler.schedulers.background import BackgroundScheduler as B; print(B()._job_defaults)"
# {'misfire_grace_time': 1, 'coalesce': True, 'max_instances': 1}
I ran this on 3.11.1 and 3.11.3. The values are identical in both.
Default 1: misfire_grace_time = 1 second
The docs describe it as "seconds after the designated runtime that the job is still allowed to be run". They do not say what a default of 1 second implies. A date job scheduled 30 seconds in the past never ran, and it disappeared from get_jobs() rather than waiting for a retry. All that was left:
WARNING apscheduler.executors.default Run time of job "job_a (trigger: date[2026-11-01 21:50:08 CST], next run at: 2026-11-01 21:50:08 CST)" was missed by 0:00:30.006098
Adding misfire_grace_time=3600 made it run. One second is fine for "fire every second inside a live process". It is useless after a laptop suspend or a container restart.
Default 2: coalesce = True
The user guide says: "if coalescing is enabled for the job and the scheduler sees one or more queued executions for the job, it will only trigger it once." That sentence gets read too broadly. It applies within a single wakeup, and the implementation is one slice:
run_times = (run_times[-1:] if run_times and job.coalesce else run_times)
A pure-function measurement avoids scheduler jitter: interval of 2 seconds, next_run_time at t0, now = t0 + 11s. Job._get_run_times() returns 6 trigger points; coalesce=True submits 1, coalesce=False submits 6.
A real pause/resume test agrees: after pausing for 11 seconds, coalesce=True caught up 2 times and coalesce=False caught up 6 times. So "how many catch-up runs" is neither 5 nor 1. It depends on which window you count, which means you cannot use it as a fixed alert threshold.
Default 3: max_instances = 1
Official wording: "By default, only one instance of each job is allowed to be run at the same time." Measured: 1-second interval, 3 seconds of work per run. In 8 seconds the job ran twice and emitted six lines like this:
WARNING apscheduler.scheduler Execution of job "slow (trigger: interval[0:00:01], next run at: 2026-11-01 21:51:12 CST)" skipped: maximum number of running instances reached (1)
It is a WARNING, easy to drown out. What it means is "your job is slower than its interval". What people conclude is "jobs are being dropped".
The fourth trap: the main thread exits
Not a default, but more common than the other three. BackgroundScheduler is background only relative to a blocking scheduler; it is still an in-process thread. Script exits after 1 second, job due at 3 seconds:
- child process returns
rc=0, prints nothing - 4 seconds later the marker file does not exist
sched.shutdown(wait=True)does not help; it waits for running jobs, not for jobs that have not fired
If that script is a docker run or a CI step, you get a green check and a job that never happened.
Celery: three boundaries
Celery usually fails at startup, but the error looks like an environment problem rather than a configuration one.
Boundary 1: memory:// is threads only
kombu's docs for the in-memory transport are one sentence:
> Simple transport using memory for storing messages. Messages can be passed only between threads.
I tested it literally across processes: the producer's send_task() returned a task id, and the consumer's drain_events() timed out with TimeoutError. The message never left the first process's heap. Testing with memory:// proves your function can be called, not that your queue connects.
Boundary 2: the first error when no broker is configured
Without a broker argument, app.conf.broker_url is None, but the connection does not fail on that. It falls back to the AMQP default and returns something that looks unrelated:
kombu.exceptions.OperationalError: [WinError 10061] No connection could be made because the target machine actively refused it.
On Windows that 10061 almost always means "nothing is listening on 5672", not a firewall and not credentials. Print the resolved broker_url before editing config.
Boundary 3: eager mode does not raise
task_always_eager=True is the usual way to skip the broker in unit tests. It does not change task_eager_propagates, which defaults to False, so a failing eager task raises nothing:
app.conf.task_always_eager = True
r = bad.delay() # no exception
print(r.state) # FAILURE
print(r.result) # eager boom
r.get() # raises ValueError here
One line in the source: throw = app.conf.task_eager_propagates. So "eager tests are green" only proves the task was called. Either set task_eager_propagates = True, or assert r.state == 'SUCCESS'.
On versions: the box already had Celery 5.3.1; PyPI currently ships 5.6.3 (2026-03-26). Both defaults are the same in either version, but 5.3.1 is off the maintenance line, so prefer 5.6.x for Python 3.13.
Troubleshooting: five real errors
Every error below was reproduced on my machine. None are copied examples.
Error 1: sqlite3.OperationalError: database is locked
Symptom: several processes enqueue into the same SqliteHuey file; some exit with rc=1; the queue holds fewer tasks than expected.
Cause: SqliteStorage._set_journal_mode() runs pragma journal_mode="wal" on a new database, which upgrades a read lock to a write lock. Under concurrency that fails, and the retry window is timeout.
Fix: do not shrink timeout. Shard the database or funnel writes through one process.
# Wrong: timeout treated as a performance knob
h = SqliteHuey('q', filename='q.db', timeout=0)
# Right: keep the default 5 seconds, raise it for heavier write concurrency
h = SqliteHuey('q', filename='q.db', timeout=5)
# Reproduce: 3 processes x 400 tasks, then count what landed
for i in 1 2 3; do python enqueue_worker.py p$i 400 & done; wait
python -c "import sqlite3;print(sqlite3.connect('q.db').execute('select count(*) from task').fetchone())"
Error 2: HueyException: not found in TaskRegistry
Symptom: the consumer logs repeated Error reading from queue tracebacks while the queue length falls.
Cause: dequeue() pops the message before deserializing it. An unregistered name raises after the message is gone.
Fix: keep an alias for the old name and remove it only after the queue drains.
@h.task(name='legacy_name') # old name stays consumable
def new_name(x):
return x
Error 3: Run time of job ... was missed by 0:00:30.006098
Symptom: a date job never runs and is absent from get_jobs().
Cause: misfire_grace_time defaults to 1 second; anything later is treated as missed and removed.
Fix: set it explicitly to match your tolerance.
sched.add_job(job, 'date', run_date=when, misfire_grace_time=3600)
Error 4: Execution of job ... skipped: maximum number of running instances reached (1)
Symptom: an interval job runs far less often than expected and the WARNING repeats.
Cause: max_instances defaults to 1; the previous run has not finished, so this one is skipped, not queued.
Fix: confirm whether the job is genuinely slower than its interval, then raise the limit if overlap is acceptable.
sched.add_job(slow, 'interval', seconds=1, max_instances=3, coalesce=False)
Error 5: kombu.exceptions.OperationalError: [WinError 10061]
Symptom: task.delay() raises a refused connection.
Cause: no broker configured, so the client falls back to localhost:5672.
Fix: configure and verify before deploying.
python -c "from celery import Celery; print(Celery('x').conf.broker_url)"
set CELERY_BROKER_URL=redis://127.0.0.1:6379/0
Choosing on one machine
Folding the boundaries above into a decision tree, order matters:
1. You only need "run once in N seconds" and the process lives forever -> skip the queue, use threading.Timer or the OS scheduler.
2. Single machine, need persistence and retries, one extra process is fine -> SqliteHuey. No extra service, but you own the consumer.
3. Single machine, short jobs, no extra process -> APScheduler, with all three defaults written out explicitly.
4. Multiple machines, burst smoothing, Redis or RabbitMQ already present -> Celery. Without a broker you cannot even observe its real behaviour.
5. Short-lived processes (CLI, CI, serverless) -> none of them; use event triggers.
The practical version: work out how long your process lives first. If the process dies before the interval, every in-process scheduler is a no-op.
A minimal hardening checklist
- huey: consumer runs as its own supervised process; task names are a contract; do not treat
Resultas a database (preserve=Truewhen re-reading); keepSqliteHueytimeoutat 5 or above. - APScheduler: write
misfire_grace_time,coalesceandmax_instancesinto the code; with a persistent jobstore always passidplusreplace_existing=True; keep the main thread alive until jobs finish. - Celery:
memory://for unit tests only; printbroker_url; assertstatein eager tests; enabletask_eager_propagateswhere you need it.
For the same failure pattern in another corner, subprocess: seven silent failures behind exit code 0 covers processes that exit 0 with no output, and n8n Task Runner external and queue mode describes what happens once the executor moves out of the main process.
FAQ
Q: huey's Result() returns None. Did the task fail?
No. None means the result is not in the store yet, because nothing consumed it or it is still running. A genuinely failed task raises TaskException when you read it, with the original exception string and traceback.
Q: Why does my APScheduler job run once and stop?
Check the trigger first. A date trigger only fires once and is removed afterwards. For interval, look for the max_instances WARNING and compare the job's duration with its interval.
Q: I tested Celery with memory:// and it worked. Can I ship it?
No. kombu states that messages can only pass between threads. Another process cannot receive them, and another machine certainly cannot. Run the real path against Redis or RabbitMQ before shipping.
Q: Can these three be combined?
Yes, but there is little reason to. The common combination is APScheduler for timing plus Celery for execution. Then APScheduler's misfire logic decides whether a task is dropped, and Celery's ack logic decides whether a dropped task comes back. Debugging two semantics at once is worse than picking one.
Closing
The three failure modes are side effects of three design philosophies. huey hands execution to another process, so by default nothing executes. APScheduler values punctuality over catch-up, so it waits one second by default. Celery hands delivery to a broker, so without a broker it will not even tell you it is empty.
Next time a job "did not run", check in this order: queue_size() for huey, _job_defaults for APScheduler, broker_url for Celery. Each is one command.
👉 Join Xiaomi MiMo Platform: Leading AI model platform with cost-effective inference
👉 Join Aliyun AI: Top AI products with exclusive coupons for business innovation
📌 This article was AI-assisted generated and human-reviewed | TechPassive — An AI-driven content testing site focused on real tool reviews
🔗 Recommended Tools
These are carefully selected tools. Using our affiliate links supports us to keep producing quality content:
As an Amazon Associate, I earn from qualifying purchases at no extra cost to you.