Prometheus Quantiles: Default Buckets Miss p90 by Half

That p99 on your dashboard is not a measurement. It is an estimate produced by a bucket layout somebody chose before the data arrived, plus a linear interpolation performed on the server. Get the buckets wrong and the number stays wrong forever, with the same error, night after night.

I pushed one million synthetic latency samples through prometheus_client's default buckets and recomputed the quantiles locally using the logic Prometheus uses. The result was not encouraging: p90 came back 49.7% low, and p999 was pinned at exactly 10 seconds. Below are seven metric semantics that look right and are not, plus five real error messages from this machine.

TL;DR: quantiles are estimates, not statistics, and their error is set by bucket width. The default buckets only fit services where most requests finish under 100 ms. Never average quantiles across instances, sum the bucket counts instead. The Python client's Summary does not export quantiles at all. No affiliate links in this article.

Test setup and what is actually measured

Environment: Windows 11, CPython 3.13.12, prometheus-client 0.26.0 installed into an isolated directory so the main environment stayed untouched. The sample set is one million synthetic latencies in three tiers: 90% between 20 and 80 ms, 9% between 200 and 800 ms, and 1% from 1 to 20 seconds as a long tail.

The real histogram_quantile lives on the Prometheus server, written in Go in promql/quantile.go. There is no Prometheus instance on this machine, so I reimplemented its bucketQuantile logic in Python: sort by le, find the bucket whose cumulative count crosses q × total, interpolate linearly inside it, and if the crossing point lands in the +Inf bucket, return the previous bucket's upper bound.

One scope note, stated up front: the interpolation is my reimplementation, not a measurement against a running server. Bucket boundaries, bucket counts, defaults and error texts all come from real execution here. The two should agree, but run the actual PromQL if you want to verify on your own cluster.

Why a quantile is computed, not counted

The official Prometheus documentation is unusually direct about this:

Quantiles, whether calculated by the instrumented binary or on the Prometheus server, are estimated. It is important to understand the errors of that estimation.

In the comparison table on the same page, the quantile error row for classic histograms says only this:

Error is limited by the width of the bucket the quantile is located in.

Read that again. The accuracy ceiling of the p99 you see is decided by how wide the bucket is that p99 happens to fall into. It is not decided by how many samples you collect. A hundred million samples will not help; the bucket width does not change.

Number one: p90 reported 49.7% low

Start with the defaults. In prometheus_client's Histogram constructor, the buckets argument defaults to fifteen boundaries plus infinity:

0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1.0, 2.5, 5.0, 7.5, 10.0, +Inf

These are seconds. The first boundary is 5 ms and the last finite boundary is 10 s.

After a million observations, truth against estimate:

Quantile	True	Estimated	Error
p50	0.0534 s	0.0534 s	-0.0%
p90	0.2003 s	0.1008 s	-49.7%
p99	0.7991 s	0.9955 s	+24.6%
p999	18.0743 s	10.0000 s	-44.7%

p50 is reassuringly accurate, and that is exactly the trap. It falls inside the narrow 0.05–0.075 bucket, 25 ms wide, and the true value of 53.4 ms sits near the middle, so interpolation lands almost on truth. You glance at the dashboard, the median checks out, and you trust the whole panel.

p90 is off by nearly half. The cause is visible in the cumulative counts: le="0.1" already holds 899,959 observations (89.99%), and le="0.25" holds 907,429. The rank for p90 is 900,000, which lands in the (0.1, 0.25] bucket, 150 ms wide, and it is only the 41st observation out of the 7,470 in that bucket. Linear interpolation assumes a uniform spread inside the bucket, so it places the value 0.8 ms past the left edge and reports 100.8 ms. The true value is 200.3 ms, because the real data has a step there — that is where the 9% of slow requests begin.

Move the p90 rank by 8 ms and the estimate jumps by a 150 ms step. The documentation has a sentence for precisely this:

With a broad distribution, small changes in φ result in large deviations in the observed value.

The +Inf bucket: p999 can never tell the truth

Of the one million samples, 5,119 exceeded 10 seconds (0.51%), and every one of them landed in +Inf. Cumulative count at le="10.0" is 994,881; at +Inf it is 1,000,000.

The p999 rank is 999,000, which sits inside the +Inf bucket. That bucket has no upper bound, so interpolation has nowhere to go. The implementation's answer is to return the previous bucket's upper bound: 10.0.

So whether the true p999 is 12 seconds, 18 seconds or three minutes, the dashboard says 10 seconds. This is not an accuracy problem, it is data loss. The most expensive requests in your long tail get replaced by a constant. The documentation buries the general rule next to a sentence about returning nothing: if the histogram has no bucket with the required boundary, no interpolation happens at all, and you either get a truncated number or no result.

Narrower buckets are not automatically better

The natural reaction to the table above is to add buckets. Hold on — the second experiment shows why that alone does not save you.

This time: 100,000 samples all between 4.90 and 4.99 seconds (a batch job with a fixed runtime), buckets set to [0, 0.1, 0.5, 1, 5, 10, +Inf].

Quantile	True	Estimated	Error
p50	4.945 s	3.000 s	-39.3%
p90	4.981 s	4.600 s	-7.6%
p99	4.989 s	4.960 s	-0.6%

Every sample sits inside the (1, 5] bucket, four seconds wide. Interpolation spreads the rank evenly across the whole 1-to-5-second range, so the median comes out as 3 seconds while the real data never goes below 4.9.

Note that p99 is only 0.6% off, because its rank is already near the bucket's upper edge, and so is the interpolated point. On the same data set, different quantiles can fail in opposite directions. That is how someone ends up calibrating against p99, unaware that p50 is off by 39%.

The conclusion is narrow: bucket boundaries have to land on the actual steps in your distribution. Bucket width by itself is not a measure of accuracy. The official phrasing for this is that "picking the right buckets in advance can be challenging".

Across instances: quantiles cannot be averaged, buckets can be added

Eight replicas, a dashboard line reading avg(rate(...)) — the third experiment targets that habit.

Two instances share one bucket layout. A gets the mixed distribution and estimates p99 at 1.5741 s. B gets only 2.0–2.5 s requests and estimates 2.4850 s. The average of the two is 2.0295 s.

The true p99 of the merged sample is 2.4949 s.

• Averaged quantiles: 2.0295 s, error -18.7%
• Bucket counts summed, then quantiled: 2.4843 s, error -0.4%

The reason is plain: a quantile is a position, not an additive quantity. If each replica serves 10,000 requests, p99 sits at rank 9,900. Averaging the two replicas' p99 values assumes that rank 9,900 in the fast replica and rank 9,900 in the slow replica occupy the same position in the merged sample. They do not, because the slow replica's requests push the whole distribution upward.

The documentation does not hedge:

However, aggregating the precomputed quantiles from a summary rarely makes sense. In this particular case, averaging the quantiles yields statistically nonsensical values.

It even ships the counter-example as a code line:

avg(http_request_duration_seconds{quantile="0.95"}) // BAD!

Histograms are immune, because bucket counts add: eight replicas' le="0.25" summed is still "the number of requests that finished within 250 ms". Interpolate after the sum and the position is correct. This is why a wrong p99 is usually a query-writing problem, not a client-library problem.

The Python Summary does not export quantiles

The concepts page defines Summary as computing configurable quantiles over a sliding time window, and lists the exposed series as {quantile="<φ>"}. Follow that description in Python and you hit a wall.

What actually comes out is three series: req_seconds_count, req_seconds_sum and req_seconds_created. Try to ask for quantiles:

Summary('req2', 'doc', quantiles=[0.5])
TypeError: MetricWrapperBase.__init__() got an unexpected keyword argument 'quantiles'

The constructor signature has no such parameter at all (name, documentation, labelnames, namespace, subsystem, unit, registry). The Python client documentation states it plainly:

The Python client does not compute quantiles locally. If you need p50/p95/p99, use a Histogram instead.

So there is a real gap: Go and Java Summaries compute quantiles, Python's does not. The same metric design document changes meaning when it crosses a language boundary. If you use Summary in Python, all you actually get is a count and a sum — that is, a request count and an average latency, with no quantiles. And the average is the least informative number in any long-tailed distribution.

Label cardinality: 5,000 combinations, 90,000 series

The fourth experiment targets the habit of tagging user_id because it seemed useful at the time.

One Histogram, two labels: user_id with 1,000 values and path with 5.

1000 × 5 = 5,000 label combinations

Exported series count: 90,000.

That is 18 series per combination: fifteen _bucket series including +Inf, plus _sum, _count and _created. Multiply your label combinations by 18 and you have both the number of lines your scrape endpoint has to emit and the number of time series your storage has to index separately.

A single user_id label inflates the metric by a factor of 1,000 while barely helping anyone debug. You want to know which class of user is slow, not that user 4,183 is slow. Cardinality explosions do not throw errors. They show up as memory, disk and query latency all degrading together, and by the time you notice, weeks have passed.

Troubleshooting: five real errors, verbatim

Every message below came from this machine. None are paraphrased.

Error one: duplicate registration

Counter('jobs_total', 'doc', registry=reg)
Counter('jobs_total', 'doc', registry=reg)
DuplicateTimeseries: Duplicated timeseries in CollectorRegistry: {'jobs_total', 'jobs', 'jobs_created'}

Note that three names are listed. One Counter occupies three series names, so collision detection looks at the union.

The subtler variant: Counter('jobs') and Counter('jobs_total') have different names but both export as jobs_total, so they collide in the same registry too. The client appends _total automatically; write it yourself and you get a metric that looks distinct and is not.

Error two: incrementing a counter by a negative amount

c.inc(-1)
ValueError: Counters can only be incremented by non-negative amounts.

This one is helpful — the library stops you. The dangerous variant is expressing a counter as a Gauge, which accepts negatives and arbitrary jumps, so resets, wraparound and concurrent overwrites all pass silently.

Errors three and four: missing and extra labels give the same message

c.labels(method='GET')                          # missing code
ValueError: Incorrect label names

c.labels(method='GET', code='200', extra='x')   # extra label
ValueError: Incorrect label names

Both cases produce identical text, so the log cannot tell you whether you omitted a label or added one. Print labelnames and diff it; do not try to diagnose from the message.

Error five: passing quantiles to Summary

TypeError: MetricWrapperBase.__init__() got an unexpected keyword argument 'quantiles'

This is the runtime form of the gap described earlier. At least it crashes instead of degrading silently.

The silent one: reset on restart

A Counter may only increase, or reset to zero on restart. After a process restart the same series drops from 500 to 3:

before restart: http_requests_total 500.0
after restart:  http_requests_total 3.0
raw difference: -497

Subtracting raw values yields a negative rate. Prometheus's rate() and increase() handle the reset by splitting the increments around the reset point, but scripts and dashboards that aggregate outside Prometheus do not. Anywhere you subtract Counter values by hand, ask whether that code knows about resets.

There is also a _created series holding a creation timestamp (measured here as 1.793312098127713e+09). One of its jobs is to let the server recognise a restart so the reset is handled correctly.

One more behaviour worth knowing: a callback registered with Gauge.set_function(fn) runs once per scrape. After three calls to generate_latest() the callback counter read exactly 3. If that function queries a database, your scrape interval is now your database QPS.

A bucket checklist you can copy

1. Plot the distribution before choosing buckets. Take a week of real latency data on a log scale and put boundaries on the steps, not on the defaults.

2. Keep one exact boundary for your SLO. If the SLO is 95% of requests within 300 ms, you must have le="0.3". Official wording: without that boundary no interpolation happens and you get no result, or an incomplete result with no warning at all.

3. Before trusting p999, check the +Inf share. (count{le="+Inf"} - count{le=""}) / count{le="+Inf"} is the fraction that proves your quantile has been truncated. Here it was 0.51%, and p999 was already wrong.

4. Use sum by (le) across instances, never avg. Aggregate the buckets first, then let histogram_quantile do its job.

5. Keep labels low-cardinality. Method, route template, status code are fine. User ID, request ID, raw URL are not. 5,000 combinations is 90,000 series.

6. Do not expect quantiles from a Python Summary. It gives you count and sum.

7. Do not hide scrape-rate-sensitive work inside set_function.

If you run an inference server that ships its own metrics

Many inference frameworks expose Prometheus metrics directly, and the bucket boundaries are the framework author's choice, not yours. I measured block-allocation overhead in the vLLM 0.31 KV cache writeup, but time-to-first-token and per-token latency need the same bucket review before you trust them. The framework hands you the default buckets, and the default buckets assume most requests finish under 100 ms — an assumption that rarely holds for inference.

The same instinct applies on the logging side. If the metric claims p99 is 10 seconds and the logs show no such request, suspect the metric before the logs. The Python logging piece covers logs that get lost, and the subprocess piece covers exit code 0 with nothing behind it. Read together they are three faces of one problem: every number an observability tool hands you was computed by that tool, and none of them is the fact itself.

Closing

A quantile is not a number, it is an estimate with an error bar attached. The p99 on your panel rests on at least four assumptions: bucket width, the uniform-distribution assumption inside the bucket, the overflow share in the +Inf bucket, and the aggregation method. The first three are decided in the client, the fourth in your PromQL. When all four hold, the number is worth using for capacity decisions. When any one of them fails, you are reading a constant dressed up as a measurement.

👉 Join Xiaomi MiMo Platform: Leading AI model platform with cost-effective inference

👉 Join Aliyun AI: Top AI products with exclusive coupons for business innovation

📌 This article was AI-assisted generated and human-reviewed | TechPassive — An AI-driven content testing site focused on real tool reviews