Blog
When your enrichment provider goes dark
A vendor's failure rate went 0.3%, 7.1%, 45.6%, 82.9%, 98.1%, then 100% over six weeks. We read it as our own bug the whole time. What we should have been watching.
An API rarely dies at a moment you can point to. Ours died over six weeks, and we spent most of them debugging ourselves.
The slide
Profile-call failure rate, by week, from our own logs:
| Week of | Failure rate |
|---|---|
| 14 Jun | 0.3% |
| 28 Jun | 7.1% |
| 5 Jul | 45.6% |
| 12 Jul | 82.9% |
| 26 Jul | 98.1% |
| 28 Jul | 99.1% |
| 29 Jul | 100.0% |
The last successful call of any kind was 29 July. By 31 July both endpoints returned 503 with
the plain-text body no available server, with or without a valid API key, which ruled out
credentials and billing.
DNS still resolved. Something answered in under 100ms. That is the signature of a proxy or load balancer with no healthy backends behind it, and it is indistinguishable from a live service right up until you read the body.
Why six weeks
Three reasons, all ours.
Our failure metric was one bucket. Everything that was not a 404 or a 429 was logged as
failed. A 503 and a malformed request looked identical in our dashboards, so a rising failure
rate read as “our data got worse”, not “the service is dying”.
7.1% is inside the range of a normal bad week. The second data point is the one that mattered and the one that was easiest to explain away. By the time 45.6% made it undeniable, we had lost a month.
We had no synthetic check. Every signal we had was mixed in with real traffic, whose composition changes for its own reasons. One scheduled call, with a known-good identifier, on a fixed cadence, would have isolated the vendor from our own data within days.
What we lost, and what we did not
We lost the ability to enrich. We did not lose the records we had already retrieved, because we had stored raw payloads rather than only the fields we parsed at the time. When the vendor took its public documentation offline, we were able to reconstruct a full field reference from 1,000 stored person responses and 500 company responses, every field verified against real traffic.
That is the entire argument for keeping raw payloads, and it is worth more than the storage costs. A stored response is the only asset that survives a vendor.
What we changed structurally
No single source is load-bearing. This is now the core of how Triguna assembles a record: several independent sources are queried, and what matches is merged by documented precedence. If one is unavailable, the record still returns, with fewer fields, and with its provenance intact so you can tell the difference between “no data” and “one source is down”.
Provenance is not a nice-to-have. During the slide we could not answer “which parts of our
stored data came from the failing path” because we had not recorded where anything came from.
Every record we return now carries source and fetched_at for exactly this reason.
Depend on shapes, not vendors. Source identifiers in our API are opaque and stable. You can compare and audit records without your code depending on which upstream produced them, which means an upstream can be replaced without breaking your joins.
If you are integrating anything today
- Store raw payloads, not just parsed fields.
- Log the status code and the error body. Always.
- Alert on a change in failure rate, not a threshold. 0.3% → 7.1% is a twenty-fold move that no fixed threshold would have caught.
- Run a synthetic check per dependency, with a known-good input.
- Ask what happens to your product when the dependency returns
503for a month. If the answer is “we find out slowly”, fix that before you fix anything else.
Related: what 130,220 calls taught us and provenance.