Store the raw payload, not just the fields you parse today

Most enrichment integrations parse six fields out of a hundred-field response and throw the rest away. It feels tidy. It is the most expensive small decision in the whole integration.

The argument in one paragraph

Every call is billed on success. A parsed row costs the same as a raw row plus a JSONB column. If next quarter you need a field you did not map, a stored payload gives it to you for free and a discarded one makes you pay for the entire corpus again, at a moment when you also have to schedule, pace, and monitor a full re-fetch.

It is not hypothetical

Two sections of our own field documentation, courses and patents, exist only because we had kept raw responses. Nobody had mapped either field. When we came to write the reference we recovered them by querying 1,000 stored person responses and 500 company responses, and every field we documented was verified against real traffic rather than a vendor contract.

The shape

One column. That is the whole change.

alter table people add column raw jsonb not null default '{}'::jsonb;

-- Index only what you query. The rest is archive, not a query surface.
create index on people ((raw->>'public_identifier'));

Two rules keep it from becoming a mess:

1. The parsed columns are the contract. Application code reads people.current_title, never raw->'experience'->0->.... Reaching into the blob from business logic recreates every problem the parsed columns exist to solve.
2. raw is for backfills and audits. When you need a new field, you write a migration that populates a new parsed column from raw, then application code reads the column.

What it costs

A person response with all optional sections is a few kilobytes of JSON, and JSONB compresses well. A million records is single-digit gigabytes, less than the price of re-fetching a million records by orders of magnitude, and you already paid for the data once.

If storage is genuinely the constraint, compress and cold-store payloads older than your refresh window rather than discarding them. Cold bytes still beat a re-fetch.

The audit case

The other reason is provenance. A record carries source and fetched_at; the raw payload carries the state those fields describe. When someone asks in six months why an account was routed to enterprise sales, “here is the exact response we received, on this date, from these sources” is a complete answer. A parsed row with three columns is not.

We wrote more about that in provenance, and about what happens when you do not have it in when your enrichment provider goes dark.

The rule

Parse what you query. Store what you received. They are different jobs, and only one of them is recoverable later.

data-modellingcostpractices