AI shipment webhooks: handle late and duplicate events

An operations assistant sees “in transit,” then a late webhook says “picked up.” If the application replaces shipment status with the last message received, the assistant can confidently explain a backward move that never happened. Duplicate delivery adds another problem: the same update may start the same customer task twice.

The following carrier integration is fictional. Its contract supplies a stable event ID and an increasing version for each shipment. Those are assumptions for the design, not claims about a carrier API. Before implementing it, check the actual provider’s identity, ordering, correction, authentication and replay rules.

Keep event time, arrival time and version separate

The arrival log is correct: E-17 really reached us after E-18. The shipment projection should nevertheless remain at version 18 under this assumed contract. E-17 belongs in history as late evidence. The second E-18 belongs to the existing delivery record and should not create another business action.

Stripe offers a documented example of why integrations must establish these rules. Its webhooks can arrive out of order or more than once, and event timestamps can collide. Its guide recommends tracking event IDs for duplicate deliveries. That is Stripe’s contract; it does not establish ordering guarantees for a shipment provider. Stripe: event ordering and duplicate deliveries

An event timestamp is often useful evidence, but it is a weak substitute for a provider-defined version. Clock differences, coarse timestamp precision and corrections complicate comparisons. A carrier without comparable versions may require a current-state API read and reconciliation. Preserve uncertainty if that API is unavailable; asking the language model to guess the latest status does not recover the missing contract.

Receive durably before starting the agent

I would put a small authenticated receiver in front of the model. It verifies the delivery using the provider’s supported method, checks the tenant and shipment mapping, then persists the event in a durable inbox. A unique key includes the provider account and event ID so identical-looking IDs from different accounts do not collide.

Stripe’s signature verification, for example, requires the unmodified request body. Its guidance also separates a quick endpoint response from heavier asynchronous work. Follow the particular provider’s verification procedure rather than assuming any JSON containing a familiar event name is authentic. Stripe: signature verification and asynchronous handling

In this proposed design, acknowledge an accepted delivery only after its inbox transaction commits. A successful HTTP response followed by a failed database write would lose the event locally. If storage is unavailable, use the provider’s documented failure response and retry behavior. A repeated accepted event can return the normal acknowledgement without scheduling a second copy.

Keep receipt state separate from processing state. “Stored” means the receiver preserved the event. “Applied” means a worker updated the projection. A failed worker remains visible for retry or investigation. Never mark an inbox row processed before its dependent state update has committed.

Update the projection under a concurrency guard

BEGIN TRANSACTION
  claim inbox event and lock shipment S-204
  if event already applied: finish without new work
  if source version is older: retain as late history
  if equal version conflicts: hold for reconciliation
  if newer and transition is supported:
    update projection with source version
    write any follow-on task to local outbox
  mark the inbox disposition
COMMIT

The worker needs a lock or an equivalent compare-and-swap check. Otherwise two workers can both read version 16, process versions 18 and 17 concurrently, and commit the older state last. A version comparison performed outside the protected update does not prevent that race. Keep the exact event and disposition available when an operator opens the shipment history.

A jump from version 16 to 18 needs another contract decision. If updates are full snapshots, applying 18 may be valid. If each update is a delta that depends on 17, buffer and retrieve the missing change instead. Unknown event types, conflicting equal versions and provider corrections should enter reconciliation under explicit rules.

AWS describes the transactional outbox for the case where a database change must also produce a message. Writing the state change and outbox entry in one database transaction avoids one succeeding without a local record of the other. The relay and consumer still need to handle duplicate messages. AWS: transactional outbox and duplicate consumers

For a customer-update task, I would key the intended effect by shipment, accepted source version and task type. That identity is separate from the incoming event ID. Two distinct source events might request the same effect, while a correction may require a genuinely new task. Define those cases instead of permanently suppressing every future event with the same status word.

Expire the agent proposal when the state changes

Now let E-19 report a delivery exception. The agent reads version 19 and drafts a customer update. Before a person approves it, E-20 confirms delivery and the projection advances to 20. A well-written draft based on 19 is now stale. Store its source version, evidence IDs and proposed action with the draft.

At execution, compare that basis with the current state under the operation’s concurrency control. In this example, 19 differs from 20, so hold the old proposal and prepare a fresh review item. Do not silently rewrite an approved message and send it under the earlier approval. A slow model response is subject to the same check as a draft waiting for a person.

For an external send, a local check cannot freeze the carrier’s world. Recheck immediately before dispatch, serialize related local operations where appropriate, and record what version supported the message. A provider-side conditional operation can offer stronger protection when available. Delivery can still change after dispatch; the workflow needs a correction path rather than an impossible promise of permanent freshness.

Test a trace that arrives in the wrong order

Track inbox age, unapplied events, reconciliation backlog and duplicate business effects separately. A green webhook delivery chart only shows that requests reached an endpoint. The operating test is whether the shipment record, the agent’s evidence and the eventual action agree after the interrupted trace.

Once an external action has started, its uncertain outcome needs a separate recovery path. The retry article covers operation identity and reconciliation after a timeout; the inbound inbox described here establishes which evidence was allowed to start that action. Read: recover an uncertain external action