gRPC defaults tested: 4 MiB caps and three retries
This time I tested seven of those defaults, one at a time, on a single machine: Windows 11, CPython 3.13.14, grpcio 1.85.1, grpcio-tools 1.85.1, protobuf 7.36.2, installed with pip into an isolated directory so the main environment stayed untouched. The server runs as a subprocess and the client runs in the main process. That split matters: inside one process the server counter and the client share objects, so you end up reading numbers you wrote yourself, and cancellation and deadlines are decided at the transport layer, which an in-process shortcut bypasses.
TL;DR: 4 MiB is the receive limit for a single message in each direction, and both directions block separately. Metadata was rejected at 12,570 bytes while the error text itself says 8,192. A 0.25 s deadline only makes the client give up; the server still finishes the 800 ms of work. A service config with maxAttempts 3 gets the server called three times. Any uncaught handler exception reaches the client as UNKNOWN.
Building the test bench first
The proto keeps six methods, each pointed at one default: Echo for message size and per-call overhead, BigDown for hitting the receive limit, Sleep and SleepCheck for deadlines, Fail for retry and error codes, Stream for cancellation, and Count for reading back what the server actually did.
| Method | Purpose | Default under test |
|---|---|---|
| Echo | 5,000-record round trip | message size, per-call cost |
| BigDown | return N bytes | 4 MiB receive limit |
| Sleep / SleepCheck | sleep N milliseconds | deadline and side effects |
| Fail | raise or abort by kind | error codes, retries |
| Stream | one message every 10 ms | streaming cancellation |
| Count | read back counters | what the server really did |
The server prints its port to stdout and the client reads that first line, so the two processes never fight over a fixed port:
proc = subprocess.Popen([sys.executable, "server.py"], stdout=subprocess.PIPE, text=True)
port = int(proc.stdout.readline().strip().split("=")[1])
channel = grpc.insecure_channel("127.0.0.1:%d" % port)
grpc.channel_ready_future(channel).result(timeout=10)
One calibration number before anything else: 500 small Echo calls averaged 0.709 ms per round trip. Every later claim of the form "the server was still working" sits on that scale, and a 0.25 s deadline leaves room for hundreds of calls, so a server that keeps running after a timeout is not a sampling artifact.
Default one: 4 MiB per message, once in each direction
Size first. Five thousand records (id, name, email, boolean, float, three tags) serialize to 322,641 bytes in protobuf and 598,430 bytes as compact JSON, so JSON is 1.855x the protobuf payload. One round trip took 7.33 ms and the server saw all 5,000 records. The ratio depends on the wire format you pick; I measured the same field set in a five-format comparison at 50,000 records and the magnitude matches.
Then the wall. The client sends one message of 4 MiB plus 10 bytes and the server refuses it:
StatusCode.RESOURCE_EXHAUSTED
SERVER: Received message larger than max (4194362 vs. 4194304)
In the other direction, the server returns 4 MiB plus 1 byte and the client refuses it:
StatusCode.RESOURCE_EXHAUSTED
Stream removed (CLIENT: Received message larger than max (4194315 vs. 4194304))
The only difference is the prefix. SERVER: means the peer rejected your request; CLIENT: means the local side rejected the peer's response. Each side has its own grpc.max_receive_message_length, so raising one leaves the other ready to fail. After raising the client option to 8 MiB, the same 4,194,305-byte response arrived intact and the length check agreed.
Two rules follow. Responses above roughly 1 MB belong in a server stream, not in one message. And if you really need to raise the cap, raise it on both sides and be honest about why a single message exceeds 4 MB, because the usual answer is a badly shaped API.
Default two: metadata has two gates and a wrong threshold in its own error
Metadata is the place people treat as a free-form pocket. It has two gates, and I bisected them to the byte:
| Custom x-pad | Total headers | Result |
|---|---|---|
| 12,017 | 12,568 | accepted |
| 12,018 | 12,570 | RESOURCE_EXHAUSTED (soft limit, text says 8192) |
| 15,900 | 16,451 | RESOURCE_EXHAUSTED (hard limit 16384) |
The error is unusually helpful and prints the byte count of every header:
received metadata size exceeds soft limit (12570 vs. 8192); :path:54B :authority:56B
:method:43B :scheme:43B content-type:60B te:42B grpc-accept-encoding:75B user-agent:92B x-pad:12037B
Here is the detail worth remembering: the error states a threshold of 8,192 and the actual trip point was 12,570, a factor of 1.53. I could not explain that gap from the source, so I am recording it as a measurement rather than a mechanism. The practical conclusion does not change: design custom metadata to stay under 8 KB and do not probe the ceiling. The seven built-in HTTP/2 headers cost about 551 bytes, and that part is not yours to control.
Services that stuff a JWT into metadata should care about this. When the token grows, the failure is RESOURCE_EXHAUSTED, which reads like "the server ran out of resources" while the truth is "your headers are too large".
Default three: a deadline is a client promise, not a server stop button
This one has the highest cost of the seven. Take an 800 ms method and call it with timeout=0.2:
- The client returns DEADLINE_EXCEEDED after 0.2 s with details
Deadline Exceeded. - At that moment the server counters read started=1, finished=0.
- 1.2 s later, finished=1, and the handler's
context.is_active()check also counted one cancellation.
So the server knew at 0.2 s that the call had been cancelled, yet the time.sleep(0.8) inside the handler was not interrupted. It slept the full duration, returned, and the return value was thrown away. The official docs say this plainly, and two sentences are worth memorizing:
A gRPC server deals with this situation by automatically cancelling a call (CANCELLED status) once a deadline set by the client has passed.
Please note that the server application is responsible for stopping any activity it has spawned to service the RPC.
The status codes page adds the sentence that actually bites:
For operations that change the state of the system, this error may be returned even if the operation has completed successfully.
In plain terms: a client timeout does not mean the server did not finish. Any handler that writes rows, publishes messages, or decrements stock has to check context.is_active() itself, or pass the remaining time downstream. Otherwise "client retries after timeout" multiplies with "server keeps running" and you get duplicate side effects. That is exactly where my two duplicate rows came from on day one.
There is also a precondition worth stating: the Deadlines guide opens with "By default, gRPC does not set a deadline", so a call without a timeout can wait indefinitely.
Default four: retries are already on, and one call hit the server three times
I gave the Fail method a service config with maxAttempts: 3 and retryableStatusCodes: ["UNAVAILABLE"], and made the handler increment a counter on entry. The client called once. The server counter went up by 3.
The more interesting control: adding grpc.enable_retries=1 changed nothing, still 3. So this is not a switch-you-forgot-to-flip trap; retries were already enabled, and all that was missing was a policy. The Retry guide opens with:
Retries are enabled by default, but there is no default retry policy.
The same page carries two easily missed details. Backoff gets plus or minus 20% jitter, so an initialBackoff of 100 ms actually lands between 80 and 120 ms. And transparent retry exists: an RPC that never left the client may be retried without limit, while one that reached the server library but was never seen by application code gets a single retry.
The danger is multiplication. I measured tenacity at 3 attempts wrapped around urllib3 at 4 in an HTTP-layer retry test, which produced 12 requests; add a gRPC retry policy of 3 on top of that chain and the worst case is 36 requests landing on the server. Keep retryableStatusCodes for idempotent calls only, and keep exactly one retry layer in the whole chain.
Log reading needs a new habit too: three server log lines for one client call means two retries, not three client requests.
Default five: any server exception becomes UNKNOWN
Same Fail method, two implementations, two different codes on the client.
A plain Python exception (raise ValueError("record 42 not found")):
StatusCode.UNKNOWN
Exception calling application: record 42 not found
An explicit context.abort(grpc.StatusCode.NOT_FOUND, "record 42 not found"):
StatusCode.NOT_FOUND
record 42 not found
The reason is in the source: the handler wrapper in grpc/_server.py catches Exception and always reports cygrpc.StatusCode.unknown, formatting details as Exception calling application: {}. The same file has another line easy to trip over: calling abort() with StatusCode.OK is rewritten to UNKNOWN with the details blanked, leaving only an error line in the server log.
One counterintuitive check: calling a method that does not exist returns UNIMPLEMENTED with details Method not found!. That is not a 404 and must not be read as "record missing". The status codes page lists seven codes that are never generated by the library, only by user code: INVALID_ARGUMENT, NOT_FOUND, ALREADY_EXISTS, FAILED_PRECONDITION, ABORTED, OUT_OF_RANGE, DATA_LOSS. So a NOT_FOUND in production came from your business logic, not from the framework.
For monitoring, group metrics by code and watch the UNKNOWN share. That percentage is the fraction of handler exceptions nobody handled, and when it climbs something is failing silently.
Default six: the ceiling is the server thread pool, not the channel
The server runs ThreadPoolExecutor(max_workers=64) and the method sleeps 50 ms. Three concurrency levels over a single client channel:
| Concurrent calls | Elapsed | Effective rounds |
|---|---|---|
| 64 | 0.084 s | 1.68 |
| 128 | 0.127 s | 2.55 |
| 256 | 0.227 s | 4.54 |
Effective rounds is elapsed time divided by 50 ms. Sixty-four calls at 1.68 rounds means genuine parallelism; 128 at 2.55 and 256 at 4.54 is the shape of "everything past 64 is queuing". The ceiling is the server pool size, not the number of client channels.
The channel side confirms it: 200 calls of 50 ms took 0.215 s over one channel and 0.221 s over 200 separate channels. Channels are cheap, extra ones buy nothing, and HTTP/2 multiplexing already runs concurrent streams on a single connection.
The usual order of tuning is backwards. People raise client concurrency first, when the server pool is what they should look at. In Python, a larger pool helps IO-bound handlers; for CPU-bound handlers it just hands work back to the GIL. In a four-shape concurrency comparison I measured 16 threads running a pure Python loop at 21.7% slower than serial.
Default seven: streaming cancellation lands immediately, but only for live streams
The Stream method emits one message every 10 ms for a planned 60. The client took 3 and called call.cancel(). The server's sent counter stopped at 3, and one second later it was still 3.
That works because the handler checks context.is_active() before every yield. Skip that check and the generator still runs the remaining 57 iterations, with nobody collecting the output. If those iterations are paging through a database, cancelling just burned 57 queries for nothing.
So the streaming pattern is fixed: put the is_active() check on the first line of the loop, not once at the top of the function.
The same load against HTTP/1.1 with JSON
Same machine, 200 requests of 50 ms each:
| Shape | Elapsed |
|---|---|
| gRPC, 200 concurrent on one channel | 0.215 s |
| HTTP/1.1, one new connection per call | 1.221 s |
| HTTP/1.1, serial on one connection | 10.333 s |
This measures connection models, not framework quality. gRPC runs 200 concurrent streams over one connection; HTTP/1.1 either queues them (10.3 s) or opens 200 connections (1.2 s). The HTTP side here is a Python ThreadingTCPServer, not the same class of implementation as the gRPC 64-worker pool, so I am not writing "gRPC is 5.7x faster". The honest claim is narrower: running concurrency over a single connection costs differently.
Troubleshooting: six error strings from this machine
All six were produced by this test bench. None are quoted from someone else's blog.
One. RESOURCE_EXHAUSTED: Received message larger than max (4194362 vs. 4194304)
With a SERVER: prefix, the peer refused your request. Fix: split the message, switch to streaming, or raise grpc.max_receive_message_length on the server.
Two. RESOURCE_EXHAUSTED: received metadata size exceeds soft limit (12570 vs. 8192)
Custom metadata passed roughly 12.5 KB, against a threshold the message claims is 8,192. Fix: keep large objects out of metadata and put the token in a short field or the request body.
Three. DEADLINE_EXCEEDED: Deadline Exceeded, while the server finished counter keeps climbing
A client giving up is not a server stopping. Fix: check context.is_active() in the handler, or propagate a deadline downstream.
Four. UNKNOWN: Exception calling application: Protocol message CountReply has no "inactive" field.
I hit this one for real during the run: I added a field to the proto and forgot to re-run protoc, so the client held a stale stub while the server returned the new field. Fix: keep proto files and generated stubs in the same commit, and recompile both ends. It also demonstrates default five perfectly, since a plain programming error arrives as UNKNOWN.
Five. UNIMPLEMENTED: Method not found!
The method name does not exist, or the server never registered that servicer. Fix: check the full /package.Service/Method path.
Six. UNAVAILABLE: failed to connect to all addresses; last error: UNAVAILABLE: ipv4:127.0.0.1:1: WSAGetOverlappedResult: Connection refused ... 10061
Returned in 2.01 s when nothing was listening, with the inner error embedded. Add wait_for_ready=True on the same port and the same situation becomes DEADLINE_EXCEEDED after 2.0 s. One problem, two codes, which means a monitoring rule keyed on code splits it into two buckets. Decide deliberately which bucket you want.
Where this stops
Python only. grpcio 1.85.1. The Deadlines guide states that deadline propagation is on by default in Java and Go and must be enabled explicitly in C++, so these numbers do not transfer across languages.
No TLS, no load balancing, no compression, no second machine. Everything ran on loopback with round trips around 0.05 ms, and deadlines and retries behave differently once a real network is in the path.
The keepalive group produced no conclusion: I set the client's keepalive_time_ms to 1,000 against a default server minimum of 300,000 ms and could not reproduce a GOAWAY or ENHANCE_YOUR_CALM within 45 seconds. No reproduction, no claim, just this note.
Sixteen logical cores, a 64-worker server pool. Rerun everything if your box differs.
Seven rules worth copying
- Design for a 4 MiB message cap and set the option on both sides.
- Keep custom metadata under 8 KB and out of the business of carrying large objects.
- Give every call a timeout, and check
context.is_active()inside handlers. - Use retryableStatusCodes for idempotent calls only, with one retry layer in the chain.
- Call
context.abort(code, details)instead of relying on exceptions to carry meaning. - Tune the server thread pool before touching client concurrency.
- Put the
is_active()check on the first line of any streaming loop.
No affiliate links in this article. Every number came from the machine described above, and the proto plus scripts rerun on it unchanged.
👉 Join Xiaomi MiMo Platform: Leading AI model platform with cost-effective inference
👉 Join Aliyun AI: Top AI products with exclusive coupons for business innovation
📌 This article was AI-assisted generated and human-reviewed | TechPassive — An AI-driven content testing site focused on real tool reviews
🔗 Recommended Tools
These are carefully selected tools. Using our affiliate links supports us to keep producing quality content:
As an Amazon Associate, I earn from qualifying purchases at no extra cost to you.