How to design for partial enrichment records

Every field is nullable. That is not a caveat to handle at the edges. It is the normal shape of the data, and designing around it is most of the integration work.

Four kinds of absence

They look identical in a naive client and mean completely different things:

What you see	What happened	Who fixes it
`404`	The identifier did not resolve	You: slug extraction
Field absent, section absent	You did not set the flag	You: `include_contact_info=true`
Field present, value `null`	Sources held nothing	Nobody: accept it
Record never requested	Enrichment has not run yet	Your queue

Collapse these into one “no data” state and every question about your enrichment quality becomes unanswerable. Model them separately from the start:

type Enrichment<T> =
  | { status: 'pending' }                       // never requested
  | { status: 'unresolvable'; reason: string }  // no identifier to send
  | { status: 'not_found' }                     // 404
  | { status: 'ok'; record: T; source: string; fetchedAt: Date };

Never let null reach a decision quietly

The dangerous pattern is a default that looks like a value:

// Wrong: an unknown company now scores as a 1-person company.
const score = (headcount ?? 1) > 200 ? 'enterprise' : 'smb';

// Right: unknown is its own branch, and it is visible.
const score =
  headcount == null ? 'unknown' : headcount > 200 ? 'enterprise' : 'smb';

?? 0 and ?? '' are how missing data turns into confident wrong answers. If a rule cannot express “I do not know”, it will express something else instead.

Keep null out of the UI

Users read a blank field as “this person has no job” rather than “we did not retrieve one”. Say which it is:

{role.title
  ? <dd>{role.title}</dd>
  : <dd class="unknown">Not returned <small>({fetchedAtLabel})</small></dd>}

Showing the retrieval date alongside the gap does two things: it is honest, and it tells the reader whether re-requesting might help.

Score on coverage, not just on values

Track how complete your records are, per field, over time:

select
  count(*)                                                as records,
  avg((raw->>'headline'      is not null)::int)::numeric(4,3) as headline_coverage,
  avg((raw->'contact_info'->>'email_address' is not null)::int)::numeric(4,3) as email_coverage
from people
where fetched_at > now() - interval '30 days';

Coverage moving is a signal worth alerting on. A sudden drop usually means an upstream source changed, not that your parser broke, but you will not know which unless you were measuring.

Do not backfill absence with guesses

It is tempting to fill a null headcount from a heuristic, or infer seniority from a company size. If you do, mark the field as derived and keep it in a separate column. The moment a derived value sits in the same column as a retrieved one, the record’s provenance stops meaning anything, and provenance is the part of the record you were actually paying for.

alter table companies add column headcount_derived boolean not null default false;

A checklist before you ship

Every read path has an explicit branch for absent.
No ?? default that could be mistaken for a real value.
The four absence kinds are stored distinctly.
The UI distinguishes “not returned” from “empty”.
Coverage per field is measured, not assumed.
Derived values are flagged and separable from retrieved ones.

Provenance: what source and fetched_at let you claim.
Person records: mapping the fields that are usually present.
Person profile reference: which sections need which flags.