What 130,220 enrichment API calls taught us

Before Triguna existed we were a customer of someone else’s enrichment API. Between 11 June and 29 July 2026 we made 130,220 calls against it and spent $186.05. We kept the logs. Here is what they said, and what we changed because of them.

The headline distribution

A quarter of person calls failing looks alarming. It was, but not for the reason the table suggests.

Two numbers in that table are lies

The company endpoint did not have a 0% not-found rate. Our person client mapped 404 to a distinct ProfileNotFoundError. Our company client threw a generic error for every non-OK status. So company 404s, the ones caused by passing display names instead of URL slugs, were counted as failed. That is almost certainly the bulk of that 41.5%.

The failure counts are inflated by retries. 32,423 failed calls spanned only 8,209 distinct entities: roughly four attempts each. The real failure rate per entity was a fraction of what the per-call rate implied.

If you take one thing from this post: store the status code and the response body. They cost you bytes. Not having them costs you the ability to reason at all.

Retries were the real cost centre

Four attempts per failed entity, each one a full round trip. None of them billed, because failures were not charged, but they consumed wall-clock time, worker slots, and rate-limit headroom that successful calls needed.

Worse, the retry logic did not distinguish retryable from terminal failures. A 404 caused by a bad slug was retried three times, at 5-second sleeps, before being recorded. That is fifteen seconds of nothing, per bad identifier, at scale.

The fix was three lines:

const RETRYABLE = new Set([429]);
if (!res.ok && !RETRYABLE.has(res.status)) {
  return terminal(res.status, await res.text());
}

Our p95 backfill duration dropped by roughly a third with no change to success rate.

The 404s were our bug, not their gap

1.26% of person calls returned 404. We had been reading that as “the vendor does not have this person”. It was not. Both endpoints keyed off URL slugs, and a meaningful share of our identifiers were display names, stale slugs, or URLs we had never normalized.

404s were never billed, which is why nobody noticed. Nothing on the invoice says “a fifth of your company calls resolved nothing”. We only found it by counting.

This is why we now treat not_found_rate as a first-class metric with its own alert, separate from every other failure, and why the identifiers and slugs page is one of the first things in our own docs.

What we changed

1. Log the status code and body. Always. The four-outcome enum was the root cause of every other blind spot in this list.
2. Map 404 to its own error type in every client, not just the one where somebody remembered.
3. Retry 429 only. Everything else is terminal.
4. Count entities, not attempts, when reporting failure rates.
5. Store the raw payload. Two fields in our schema documentation, courses and patents, were only recoverable later because we had kept raw responses. Re-fetching would have meant paying for the same calls twice.

The part we could not fix

None of the above prevented what happened next: the service went down and stayed down. Failure rate by week ran 0.3%, 7.1%, 45.6%, 82.9%, 98.1%, then 100%. Our logs recorded the whole slide, and we watched it for six weeks reading it as our own problem.

That is a different lesson and its own post: when your enrichment provider goes dark.

data-qualityinstrumentationpostmortem