Threads, Processes or Coroutines: One Load, Four Shapes
A colleague told me to wrap our batch script in Celery, because "50,000 tasks will never finish on one thread". I did not install anything. I did something cheaper first: I split the per-task time into waiting, computing, and everything else. Those three answers point at three completely different designs, and two of them do not need a queue at all.
What follows is the result of running one identical load under four concurrency shapes, three runs each, plus one deliberate mistake per shape. The short version: concurrency shapes overlap waiting; they do not raise throughput. A queue buys you survival across restarts and backpressure. It does not buy you speed.
No affiliate links in this post. Every number below came out of my own machine, and the benchmark script is short enough to reproduce.
Measure the bottleneck before you pick a shape
I ask three questions, in this order, and I do not skip ahead:
- First: where does a single task spend its time? Waiting on a socket, a disk or a child process is I/O, and threads or coroutines can overlap it. Pure Python arithmetic only parallelises across processes. Computation inside a C extension that releases the GIL is a third case, where threads work again.
- Second: does a task need to survive a kill? If yes, you need a queue. Anything sitting in a thread pool dies with the interpreter, with no trace and no retry.
- Third: can the producer outrun the consumer? If yes, you need a bounded queue with a blocking
put. That is a memory-safety question, not a concurrency question.
Only the last two point at a queue. The first decides thread, process, or coroutine. The most common mistake I see is stacking all three layers on one bottleneck: a pure I/O job, wrapped in a message queue, with 20 threads inside each worker, against an upstream API that allows 10 requests per second. Three layers of concurrency, one bottleneck, and the failure now happens further from where it was caused.
How I measured it
Windows 11, CPython 3.13.14, 16 logical cores. The server is a minimal HTTP handler in a separate process that sleeps 3 ms and returns 100 bytes. The client issues 1,500 requests, opening a new connection each time with no keep-alive, so connection setup and header parsing are included in the measurement — that is closer to a real scraping script than a keep-alive microbenchmark.
Running the server in a separate process matters. My first version put it in a thread of the same process, and 20 workers produced only a 4.5x speedup, because client and server were fighting over one GIL. I was measuring the interpreter arguing with itself, not client concurrency.
# server: separate process, port 18765
python bench.py serve
# client: serial | thread N | queue N | asyncio N | proc N
python bench.py thread 20
Each shape runs in its own subprocess, with tracemalloc recording the Python heap peak. tracemalloc counts Python-level allocations only; interpreter overhead and thread stacks are not in that number, so treat the peak column as relative, not as RSS.
Four shapes: speedup stops at 8 workers
Three runs each, median reported. The serial baseline is 14.547 s.
| Shape | Median wall | Requests/s | vs serial | Python heap peak |
|---|---|---|---|---|
| Serial | 14.547 s | 103.1 | 1.00x | 1.75 MB |
| ThreadPool 2 | 6.386 s | 234.9 | 2.28x | 4.41 MB |
| ThreadPool 4 | 3.131 s | 479.0 | 4.65x | 4.42 MB |
| ThreadPool 8 | 1.587 s | 945.0 | 9.17x | 4.43 MB |
| ThreadPool 16 | 1.712 s | 876.0 | 8.50x | 4.47 MB |
| ThreadPool 20 | 1.776 s | 844.7 | 8.19x | 3.93 MB |
| ThreadPool 32 | 1.687 s | 888.9 | 8.62x | 3.02 MB |
| ThreadPool 64 | 1.733 s | 865.5 | 8.39x | 3.16 MB |
| queue.Queue 20 | 1.633 s | 918.5 | 8.91x | 1.89 MB |
| asyncio 20 | 2.229 s | 672.9 | 6.53x | 3.49 MB |
| ProcessPool 20 | 11.112 s | 135.0 | 1.31x | 2.55 MB |
The column that matters is the plateau. Past 8 workers, everything from 16 to 64 lands between 1.59 s and 1.78 s, and requests per second wobble between 844 and 945. Going from 8 to 64 workers costs 8x the resources and buys a throughput difference of zero.
The ceiling is not on the client. The server is a single-threaded Python process: it accepts connections, spawns a handler thread per request, and every handler competes for the same GIL. Its practical limit is around 900 requests per second. Whether the client runs 16 workers or 64, it is waiting on the same ceiling.
That is what "concurrency overlaps waiting, it does not raise throughput" looks like when you measure it. Before adding workers, ask whether the bottleneck is on your side or on the other end. If the other end is a rate-limited API, 64 threads just make the 429s arrive sooner.
Two numbers in that table deserve their own paragraph. The queue.Queue shape (20 long-lived threads) beat ThreadPoolExecutor(20) by 8% (1.633 s vs 1.776 s), because every submit builds a Future, takes locks and signals a condition variable — 1,500 times. And asyncio came in 36% slower than the thread pool, because HTTP connection setup and parsing are still single-threaded Python work. Coroutines remove the waiting; they do not remove the parsing.
The default 20 workers are 12% slower than 8
The max_workers default is documented precisely: 3.8 changed it to min(32, os.cpu_count() + 4), and 3.13 switched os.cpu_count() to os.process_cpu_count(). On this 16-core box both formulas give 20.
import os
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor
print(os.cpu_count(), os.process_cpu_count()) # 16 16
with ThreadPoolExecutor() as ex:
print(ex._max_workers) # 20
with ProcessPoolExecutor() as ex:
print(ex._max_workers) # 16
The intent of that default is a guardrail: keep at least 5 workers for I/O, use at most 32 cores for CPU-bound work. It is not a performance optimum. On my load, the default 20 finished in 1.776 s and a hand-set 8 finished in 1.587 s — the default is 11.9% slower than the best point, with run ranges of 1.740–1.870 s and 1.565–1.613 s. Those ranges do not overlap, so this is not noise.
That is worth internalising if you write batch scripts: leaving ThreadPoolExecutor() empty means "about 10% off the optimum, and you will never know why". I start at 8 and only go up when the task has a genuinely long wait (2 seconds or more per call). Double it once, measure the wall clock, and if it does not drop, you have hit the other end's ceiling — go back down.
A process pool doing I/O is 6.3x slower
The last row of that table is the expensive lesson. Twenty processes issued the same 1,500 requests in 11.112 s, which is 1.31x over serial — 6.3x slower than the thread pool, and barely better than doing nothing.
Three costs stack up:
Startup. The default start method on Windows is spawn, so each child is a full interpreter launch that re-imports the main module. Twenty children means twenty of those.
Serialisation per task. Every submit pickles the callable and its arguments into a pipe, and the child pickles the result back. When the task itself takes 3 ms, the round trip costs more than the work.
Task granularity. 1,500 tasks across 20 processes is 75 round trips each, and Executor.map defaults to chunksize=1 on a process pool, so the fixed cost is paid 1,500 times.
I separated "startup plus serialisation" from "actual computation" with a second experiment: 400 sha256 tasks of roughly 0.9 ms each took 0.354 s serially and 4.423 s through ProcessPoolExecutor(16) — 12.5x slower. On the same machine, 16 plain Python processes each running one heavy unit finished in 1.54 s against 1.33 s for a single unit, close to 13.8x — the cores are real, and so is the pool's overhead. The problem is not multiprocessing; the problem is that the tasks are too small to amortise the pool.
My rule of thumb: under 10 ms per task, do not use a process pool; between 10 and 100 ms, measure before deciding; above 100 ms, the fixed cost finally starts to amortise. Logging hits the same wall from the other side — across processes, QueueListener needs multiprocessing.Manager().Queue(-1) instead of queue.Queue, which is exactly what that Python logging post-mortem ran into.
Whether threads help depends on the GIL, not on the label
Three loads, all "CPU-bound", 16 jobs each:
| Load | Serial | 16 threads | Result |
|---|---|---|---|
| Pure Python loop, 3M iterations | 6.103 s | 7.425 s | threads are 21.7% slower |
| hashlib, 256 KB per update | 0.018 s | 0.007 s | threads are 2.6x faster |
| hashlib, 1 KB per update | 0.020 s | 0.023 s | no speedup |
The dividing line is 2,047 bytes. From the hashlib docs: "To allow multithreading, the Python GIL is released while computing a hash supplied more than 2047 bytes of data at once in its constructor or .update method." Feed it more than 2,047 bytes at a time and the C-level work happens outside the GIL, so 16 threads really do run in parallel. Feed it 1 KB at a time and the GIL is held throughout; the threads just queue for a lock and pay for context switches.
The same reasoning extends to every C extension: parts of zlib, lzma, numpy, and the C layer under sqlite3 may release the GIL. "Python has a GIL, so threads are useless" is wrong. The accurate statement is "the GIL decides which threads are useful." And you cannot settle it by reading docs — you settle it by running the same task serially and threaded and subtracting.
The first row also proves something people find surprising: 16 threads on a pure Python loop were slower than serial, by 21.7%. That is not an anomaly; it is the cost of contending for the GIL. "More threads can never be worse" is a comfortable illusion.
What a queue is actually for: backpressure
People reach for queue.Queue to get concurrency. Its real contribution is bounding the number of in-flight tasks. 50,000 tasks, 8 consumers, 0.5 ms each:
| Shape | Time to enqueue | Total wall | Peak queued | Python heap peak |
|---|---|---|---|---|
Queue() unbounded | 0.501 s | 7.261 s | 49,882 | 1.91 MB |
Queue(1000) bounded | 6.665 s | 6.801 s | 1,000 | 0.04 MB |
50,000 bare submit calls | 3.375 s | 9.970 s | 50,000 futures | 97.58 MB |
Read the rows carefully. The unbounded queue enqueued everything in 0.5 s, which feels great, and left 49,882 tasks resident in memory. The bounded queue spent 6.665 s just enqueuing, because the producer was forced to wait for consumers — and its total wall clock was 6.3% lower. The third row is the dangerous one: submit never blocks, so 50,000 futures piled up to 97.58 MB and the whole run was the slowest of the three at 9.97 s.
That is backpressure. It is not an optimisation; it is the only way to make production follow consumption. Leave maxsize at 0 and you have switched this feature off. The code still runs. Your memory peak just becomes a number you no longer control.
The correct skeleton puts task_done() in a finally block:
import queue, threading
q = queue.Queue(maxsize=1000)
def worker():
while True:
item = q.get()
if item is None:
q.task_done()
return
try:
handle(item) # exceptions handled inside
finally:
q.task_done() # required, or join() never returns
threads = [threading.Thread(target=worker, daemon=True) for _ in range(8)]
for t in threads:
t.start()
for item in source(): # producer blocks when the queue is full
q.put(item)
q.join() # waits for every task_done
Four defaults that never warn you
All four confirmed on this machine and documented upstream. None of them will tell you when you get them wrong:
ThreadPoolExecutor()defaults tomin(32, process_cpu_count() + 4), which is 20 here, and is not necessarily the optimum.ProcessPoolExecutor()defaults toos.process_cpu_count(), which is 16 here, capped at 61 on Windows.queue.Queue()defaults tomaxsize=0; the docs say "If maxsize is less than or equal to zero, the queue size is infinite", which turns backpressure off.Executor.submit()never blocks and applies no backpressure at all.
The last one hides behind executor.map(), which looks like it waits. It does wait, but in input order: the iterator map returns yields results strictly in submission order, so one slow task holds back every result behind it. With 10 tasks whose durations descend from 1.0 s to 0.1 s, the first result from map arrived at 1.00 s and the first result from as_completed arrived at 0.10 s. If you want "handle whatever finishes first", as_completed is the only answer.
Troubleshooting: eight errors I actually hit
Every message below was copied from a real run during this experiment, not from a list of "typical errors".
Error 1: RuntimeError: cannot schedule new futures after shutdown
Symptom: a shutdown hook submits one last job and the process dies. Cause: the executor is already closed; the docs state "Calls to Executor.submit() and Executor.map() made after shutdown will raise RuntimeError". Fix: split "stop accepting" from "drain" with a boolean guard, or wrap the submission site in try/except RuntimeError and log the rejection.
Error 2: ValueError: task_done() called too many times
Symptom: the consumer calls task_done() outside the try, and a failing task runs it twice. Cause: every task_done() must match a put(); the docs say it "Raises a ValueError if called more times than there were items placed in the queue". Fix: exactly one task_done() in finally, in the same function body as the get().
Error 3: join() never returns, and nothing is logged
Symptom: q.join() hangs forever with no exception. In my reproduction the joining thread was still alive after 2 seconds, with unfinished_tasks stuck at 1. Cause: join() waits for the unfinished count to reach zero, and taking an item out does not complete it. Fix: print q.unfinished_tasks while diagnosing, then put task_done() in finally. In production, wrap join() behind a watchdog thread with a timeout instead of calling it bare.
Error 4: BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
Symptom: a worker calls os._exit(), or gets killed, and the parent blows up inside map. Cause: the pool marks itself broken when a worker dies uncleanly and never restarts it. Fix: never os._exit() inside a worker, and rebuild the executor instead of reusing a broken one. This exception subclasses RuntimeError (measured MRO: BrokenProcessPool → BrokenExecutor → RuntimeError), so a casual except RuntimeError will swallow it.
Error 5: AttributeError: Can't get local object 'f5.
Symptom: submitting a closure defined inside a function. Note it is not the "Can't pickle local object" message I expected — searching for the text I assumed would have sent me down the wrong path. Cause: a spawn child has an empty __main__ shell and cannot resolve objects defined in the parent's local scope. Fix: submit module-level names, and pass picklable arguments.
Error 6: RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.
Symptom: on Windows, missing if __name__ == "__main__": guard around pool creation. Cause: spawn re-imports the main module, so module-level code runs again in the child and the child starts building pools too. Fix: guard the entry point, and add freeze_support() if you ship a frozen executable. I ran the same broken script three times and got two different outcomes: once the parent raised this RuntimeError, twice the child died first and the parent surfaced BrokenProcessPool instead. Searching by error string leads to two different pages for one root cause.
Error 7: RuntimeError: asyncio.run() cannot be called from a running event loop
Symptom: calling asyncio.run() inside a coroutine to "just run this small blocking bit". Cause: one running loop per thread. Fix: await it, or push the blocking call through await asyncio.to_thread(fn). Watch for the companion line RuntimeWarning: coroutine 'sleep' was never awaited, which usually shows up in logs before the exception does.
Error 8: time.sleep() inside a coroutine — no error, just zero concurrency
Symptom: 20 coroutines sleeping 50 ms each should finish in about 0.05 s; measured 1.009 s. Swapping in await asyncio.sleep(0.05) gave 0.068 s, a 14.8x difference. Cause: time.sleep blocks the thread, the loop stops, and the 20 tasks serialise. Fix: use the await form, and wrap any synchronous third-party library in asyncio.to_thread. Failures like this are worse than exceptions, because the only symptom is "slow" and nothing reaches a log. The related family of cancellation traps is covered in the asyncio cancellation piece.
When a queue is genuinely the right answer
The decision tree I ended up with:
- Question 1: would a retry cause a side effect — an email, a charge, a write to an external system? Then you need idempotency keys and persistence. Ship the queue.
- Question 2: after a kill, must unfinished tasks resume? Then the queue, or at least a durable to-do table.
- Question 3: will the producer stay faster than the consumer? Then you need a bounded queue at minimum; the queue itself is optional.
- Question 4: do tasks need to run on other machines? Then the queue.
- Question 5: if all of the above are "no", the standard library is enough: thread pool for I/O starting at 8 workers, process pool for pure Python with task sizes above 100 ms, and coroutines only when you already have an async stack or need thousands of concurrent connections.
One more criterion for case 5 comes from a sentence in the docs that is easy to skip: ThreadPoolExecutor states "All threads enqueued to ThreadPoolExecutor will be joined before the interpreter can exit", followed by "it is recommended that ThreadPoolExecutor not be used for long-running tasks". Interpreter exit waits for every thread. One stuck long task can keep the whole process alive. Long tasks belong in a process pool, a queue, or behind proper cancellation. The queue-library layer — retry semantics, visibility, locking — is a separate problem, and the seven silent failures across huey, APScheduler and Celery digs into it.
Limits of this test
- Everything ran on one 16-core Windows machine where
spawnis the default start method. Linux defaults tofork, which makes process-pool startup much cheaper; do not carry the 6.3x figure over to Linux unchanged. - The server was my own single-process Python service and it is the bottleneck in this experiment. Point the client at Go or nginx and the speedup curve keeps climbing and the plateau moves right. Do not treat 8 workers as a universal optimum.
tracemallocmeasures Python-level allocation, not RSS. Thread stacks, the interpreter and C buffers are outside it.- I did not test the free-threaded 3.13 build; the GIL section would need a rerun there.
- I did not test distributed queues across machines, nor real Celery plus Redis throughput.
What I would copy into the next script
# 1) I/O-bound: thread pool, start at 8, raise only if wall clock drops
from concurrent.futures import ThreadPoolExecutor
with ThreadPoolExecutor(max_workers=8) as ex:
for r in ex.map(fetch, urls): # order matters? use map
save(r)
# 2) First finished, first handled: as_completed, never map
from concurrent.futures import as_completed
futs = {ex.submit(fetch, u): u for u in urls}
for f in as_completed(futs):
handle(futs[f], f.result())
# 3) Pure-Python CPU work: process pool, tasks above 100 ms
from concurrent.futures import ProcessPoolExecutor
def heavy(n): ... # must be a module-level function
with ProcessPoolExecutor() as ex:
list(ex.map(heavy, payload, chunksize=8)) # do not leave chunksize at 1
# 4) Producer faster than consumer: bounded queue, task_done in finally
q = queue.Queue(maxsize=1000)
Three sentences to take away: measure the bottleneck before choosing a shape; a default is a guardrail, not an optimum; a queue buys survival and backpressure, not speed. If you want the same method applied to process-level silent failures, the subprocess post-mortem from October 29 is the companion piece.
👉 Join Xiaomi MiMo Platform: Leading AI model platform with cost-effective inference
👉 Join Aliyun AI: Top AI products with exclusive coupons for business innovation
📌 This article was AI-assisted generated and human-reviewed | TechPassive — An AI-driven content testing site focused on real tool reviews
🔗 Recommended Tools
These are carefully selected tools. Using our affiliate links supports us to keep producing quality content:
As an Amazon Associate, I earn from qualifying purchases at no extra cost to you.