Rate limits that have nothing to do with your request rate

We spent a week tuning a throttle against an API’s 429 responses before noticing that request rate had nothing to do with when they fired.

The data

Four days from our call logs, against a third-party enrichment API:

Day	Calls	Rate limited	Peak calls/sec
2026-06-30	19,154	0 (0.0%)	5
2026-07-16	201	128 (63.7%)	1
2026-07-21	3,540	984 (27.8%)	2
2026-07-23	750	377 (50.3%)	2

The heaviest day ran ninety-five times the volume of the worst day, at five times the peak rate, and was never throttled once. There is no reading of this table in which throughput is the trigger.

What we had assumed

The vendor’s code comments mentioned “100 requests/second”. We had built a token bucket against that number, then lowered it repeatedly as 429s persisted. Each reduction made the backfill slower and changed nothing about the throttle rate.

Our peak observed throughput across the entire seven-week period was 5 calls/second. We never came within twenty times of the documented ceiling. The ceiling was never the constraint, and as far as we could tell was never tested by anyone.

What actually works

Honour Retry-After. It is the only signal that reflects the server’s real state. Sleep for what it says, retry a bounded number of times, and stop.

async function withRetry(fn: () => Promise<Response>, max = 2) {
  for (let attempt = 0; ; attempt++) {
    const res = await fn();
    if (res.status !== 429 || attempt >= max) return res;
    const header = Number(res.headers.get('Retry-After'));
    const base = Number.isFinite(header) && header > 0 ? header * 1000 : 5_000;
    await sleep(base + Math.random() * 500); // jitter, or the batch resynchronizes
  }
}

Add jitter. Without it, a throttled batch retries in lockstep and recreates the burst that got it throttled: the classic thundering herd, at a smaller scale than usual but with the same shape.

Pace modestly and stop tuning. A steady, unambitious rate with correct backoff finishes sooner than an aggressive rate that spends its time in retry sleeps. We settled on four concurrent workers with a 100ms gap and stopped thinking about it.

Measure the right thing

Our metric said “1.8% of calls were rate limited”. That number was a lie in a specific and misleading way: the client retried twice with sleeps before recording a rate_limited outcome, so one logged event meant three consecutive 429s. The true share of throttled requests was roughly three times the reported figure.

If your retry wrapper swallows attempts, name the metric for what it counts: rate_limited_after_retries, not rate_limited. Otherwise you will make capacity decisions against a number that is off by the retry count.

The honest summary

You cannot throttle your way out of a limit that is not about throughput. Detect it, back off on the server’s terms, jitter, bound the attempts, and put the engineering effort somewhere it changes an outcome.

rate-limitingreliabilityinstrumentation