AI model fallback: separate refusals from outages

A model fallback changes which system answers a request. For an operations team, the useful test is whether the replacement preserves the task's meaning. A supplier quote returned as a number in the wrong unit can be more troublesome than a clear failure.

This article uses a fictional quote, Q-614, to design that test. The task is to normalize a supplier's offer for internal review. It has no booking permission. The records and proposed outputs are illustrative; no model comparison or outage experiment was run.

Classify the trigger before choosing another model

Anthropic's current refusal-fallback beta handles a safety-classifier decline returned as a normal response with stop_reason set to refusal. That server-side mechanism does not handle rate limits, overload or server errors. Its supported target models and request compatibility rules also constrain the route. Log the serving model and attempt information rather than assuming the requested model answered. Claude: refusals and supported fallback behavior

Vercel AI Gateway documents a different mechanism: string entries in its models fallback list respond to execution failure. Provider routing applies within each model before the next model is tried. Its decision API has a separate conditional-fallback feature for uncertain results. Do not generalize that condition to ordinary text requests. Vercel: model fallbacks and provider routing

A refusal does not diagnose an outage, and a refusal-free answer does not establish permission. Keep an application denial in its own state. A request to disclose another customer's confidential rates should stop at access control before any model sees that material, regardless of how fallback is configured.

Carry the quote contract through the replacement

Suppose source version Q7 says: USD 240 per pallet; four pallets; fuel surcharge excluded; validity ends at 17:00 UTC. The proposed record includes source ID, currency, amount, charge basis, quantity, exclusions and expiry. It also includes unresolved fields. The schema permits an unknown value so the model can preserve missing information instead of guessing.

Both candidates could satisfy a JSON schema containing those fields. Candidate B still changes two material facts. Code can calculate a USD 960 base charge from Candidate A only after confirming that 240 is a per-pallet amount and four is the requested quantity. The result remains a base charge with an unresolved fuel amount. It is not an all-in quote.

Store the original source version with the candidate. Validate identifiers, currency and allowed units mechanically. Review the evidence for exclusions and conditions. If the source itself is ambiguous about the charge basis, preserve that ambiguity and obtain clarification. Agreement between two models is insufficient evidence for filling it in.

Qualify the entire route before an outage

I would keep a small route register per task. Each entry names the allowed model and provider, supported input types, required tools and output contract. It also records whether that destination is permitted to receive the task's data. Changing a model can change the processor or processing location; eligibility belongs in configuration before the request is transmitted.

For Q-614, an alternate route qualifies only if it can process the supplied document form, preserve the quote fields and return a review item under the same access limits. A text-only route may be suitable for a verified text extraction. It is unsuitable when the evidence depends on an image the application omitted. Converting the request needs its own tested adapter and evidence-preservation rules.

Keep an overall deadline outside nested SDK retries. A gateway may attempt more than one provider while the application sees one call. Record available attempt metadata, latency and usage so the operator can distinguish a primary response from a fallback. For costs that arrive later, retain pending exposure instead of assuming a local timeout means the provider did no work.

Keep partial output and external actions out of replay

If a response stream fails after a partial quote, mark that candidate incomplete. Do not join its first half to a replacement model's second half. Build a fresh candidate from the same authorized source set and run the full validator. Preserve the failed attempt separately for diagnosis.

This example has read and draft tools only. If another workflow permits booking, its action ledger needs an independent recovery decision. A model failure after a tool dispatch cannot establish whether the supplier accepted the booking. Reconcile the original operation before another model proposes a replacement. The retry design explains that uncertain-outcome boundary. Read: recover an uncertain external action

Evaluate continuity at the business boundary

Build a held-out set with ordinary quotes, missing currencies, conditional surcharges, conflicting versions and unreadable pages. Run each eligible route through the same checks. Also inject availability errors, refusals and interrupted streams in a controlled test harness. These are proposed tests; this article does not report their results.

Set acceptable error levels with the operating team before comparing routes. A replacement that answers more often but requires extensive correction may be a poor recovery choice. Start with internal drafts, retain a manual path, and enable only the routes whose failure cases the team can inspect and resolve.