How to run a nightly backfill without overspending

A backfill is where enrichment budgets go to die. Every call is billed on success, there is no bulk endpoint, and the naive job (“refresh everything, retry on failure”) multiplies both.

Pick candidates, do not sweep

The first and largest saving is not calling. Refresh what is stale and read:

select company_id, public_id
from companies
where fetched_at < now() - interval '90 days'
  and last_read_at > now() - interval '30 days'   -- someone actually looks at it
order by last_read_at desc
limit :budget;                                     -- a hard cap, in rows

That limit :budget is the important line. Derive it from money, not from row count:

const MONTHLY_BUDGET_CALLS = 20_000;
const nightlyBudget = Math.floor(MONTHLY_BUDGET_CALLS / 30);

A job that cannot exceed its budget by construction is a job you can leave running.

Pace, do not burst

Your account has a real, published request-per-second allowance, 10 rps by default, and every response tells you where you stand:

X-RateLimit-Limit: 10
X-RateLimit-Remaining: 3

So unlike vendors where 429 is a mystery, you can pace deliberately. Stay under the limit, watch X-RateLimit-Remaining, and treat a 429 as a bug in your pacing rather than weather.

A small fixed gap plus a concurrency cap is enough:

const CONCURRENCY = 4;
const GAP_MS = 100;

async function paced<T>(items: T[], fn: (item: T) => Promise<void>) {
  const queue = [...items];
  const workers = Array.from({ length: CONCURRENCY }, async () => {
    while (queue.length) {
      const item = queue.shift()!;
      await fn(item);
      await sleep(GAP_MS);
    }
  });
  await Promise.all(workers);
}

Retries that do not double-spend

The failure counts in a badly built backfill are dominated by retries. The same entity attempted four times looks like four failures and can bill more than once. Two rules:

1. Send a stable Idempotency-Key derived from the entity and the run, not from the attempt: backfill:2026-07-31:company:northwind.
2. Retry only 429. 401, 402, 403, and 404 are terminal. A retry loop over 402 will run all night and achieve nothing.

const RETRYABLE = new Set([429]);

if (!res.ok && !RETRYABLE.has(res.status)) {
  await recordTerminal(entity, res.status, await res.text());
  return; // stop. This will not succeed on attempt two.
}

Log outcomes you can actually read

Recording only success / failed throws away the distinctions you need. Record the status code and keep the four outcomes separate:

If a 429 outcome is only recorded after three consecutive retries, say so in the metric name. Otherwise “1.8% rate limited” quietly means “5.4% of calls were throttled”.

Guardrails worth having

• A per-key credit ceiling sized to the run. Past it, calls return 402 rather than spending: a bounded, visible failure.
• A separate key per job. Then the backfill cannot exhaust the credits your signup flow needs.
• A dry-run mode that counts candidates and prints the projected cost without calling.
• A kill switch the on-call can flip without a deploy.

After the run

Compare fetched_at distribution before and after. If the tail did not move, your candidate query is selecting the same rows every night, usually because a terminal failure is not being recorded, so the same unresolvable entities are retried forever.

Next

• Company records: refresh cadences by field.
• Conventions and errors: retry semantics in full.
• Authentication: key scoping and ceilings.