API integration error handling, written for the person who gets the email

Search this topic and you get try/catch blocks and retry decorators in whichever framework the author uses.

That is not the problem. The problem is Monday morning, when someone in operations opens an email that says 4 rows were quarantined, and has to decide what that means and whether to care.

Error handling is an operational design question that happens to be implemented in code. Here is the version that survives contact with a real business.

Alert on absence, not just on errors#

The most common integration failure is silence.

The upstream job did not run. The SFTP credentials expired. Someone changed the export schedule. Nothing errored, because nothing happened, and a pipeline that only reports on what it processed reports nothing at all.

Every scheduled integration needs a watchdog that asserts the positive: the file arrived, it arrived within the expected window, and it was roughly the size it should be. A 40 KB file where yesterday's was 4 MB is a failure even though every row in it parsed correctly.

This single check catches more real outages than every try/catch in the codebase.

Make replays safe with a stable external ID#

The most important decision in the whole integration is made before any error handling exists: what is the key.

Every write should be an upsert keyed on a stable external ID from the source system. Not the email. Not a name and postcode concatenation. An ID that does not change when someone edits the record.

With that, replaying a batch is free. You can reprocess yesterday's file at any time and the result is identical. Without it, every retry risks duplicates, and once duplicates exist the cleanup costs more than the integration did.

If the source system genuinely has no stable ID, agreeing a composite key is stage one of the engagement, not a detail to resolve later.

Three failure classes, three different responses#

Treating every failure the same is why alerting gets ignored.

Transient. Rate limit, timeout, brief outage. Retry with exponential backoff and jitter. Nobody gets told. These are normal and a system that alerts on them trains everyone to mute the channel.

Permanent for one record. A required field is empty, a date will not parse, a referenced company does not exist. Retrying will never help. Quarantine the row, keep processing the rest, and put it in the exception report. One malformed record must never stop a batch of 14,000.

Systemic. Authentication failed, the schema changed, half the batch is rejecting. Stop. Do not process a partial file. Wake somebody.

The dead letter queue holds the second and third classes, with the raw payload and the reason, so a fix can be replayed rather than retyped.

Write the exception report for operations, not for engineers#

This is where most integrations fail as products, even when the code is fine.

A stack trace in a log file is useless to the person who owns the data. What they need is a row per failure, in a format they can act on:

Row 4,182  |  Order #SO-99213
  Reason:  Company external_id 'ACME-004' not found in CRM
  Action:  Create the company, or map ACME-004 to an existing record
  Replay:  automatic on next run once resolved

Three properties make it work. The reason is in business language. The action says who does what. And the record identifies itself by something the reader recognises, an order number rather than an internal row index.

Get this right and quarantined rows get resolved by operations in ten minutes. Get it wrong and they accumulate for months until someone notices the numbers do not tie.

Publish the run summary even when nothing is wrong#

A healthy run should still produce a visible artifact:

Run 02:00 UTC
  14,208 rows read
   1,046 created
  13,158 updated
       4 quarantined  -> see exception report
  Next run 02:00 UTC

Two reasons. It gives the team a baseline, so an abnormal run is obvious. And it proves the pipeline is alive, which the absence of errors does not.

Four quarantined rows out of 14,208 is healthy. Zero quarantined rows every single day usually means validation is not actually running.

Decide who gets woken, in advance#

The runbook question that gets skipped: for each failure class, who is notified, through which channel, and what is the expected response time.

Transient failures go nowhere. Quarantines go to a daily digest for the data owner. Systemic failures go to a person, immediately, with the runbook link in the alert itself.

If the answer to "who gets paged" is "we will figure that out", the integration is not finished regardless of how well it runs.

The handover test#

An integration is complete when someone who did not build it can do four things from the runbook alone: reprocess a failed batch, interpret each quarantine reason, rotate the credentials, and know what to do when the vendor deprecates an endpoint.

Until then it is not a deliverable, it is a dependency. The same standard applies whether the pipeline is moving orders between systems or deciding which records need a human.

Frequently asked questions#

How many retries should an integration attempt?#

Three to five with exponential backoff and jitter for transient failures, then move the record to the dead letter queue. Retrying a permanent failure forever is how queues silently fill up.

What should happen when one row in a batch is invalid?#

Quarantine that row and keep processing. A single malformed record stopping a batch of thousands is the most common cause of a sync being switched off entirely, which is far worse than one bad row.

Who should receive integration alerts?#

Split them. Transient failures go nowhere, quarantined rows go to the person who owns the data in a daily digest, and systemic failures go to a named on call person immediately.

How do we know the integration is still working?#

By asserting the positive. Check that expected data arrived in the expected window at roughly the expected volume, and publish a run summary even on successful runs. Absence of errors is not evidence of operation.