LLM Provider Fallback for OpenAI, Claude, and Gemini

OpenAI starts returning 429s halfway through an otherwise quiet morning. A few calls time out. The SDK retries twice, the API handler retries on top of that, and users begin refreshing the page. Within seconds, the application is sending more traffic to the dependency that is already struggling.

This is how a routine provider incident becomes a retry storm. A better response is LLM provider fallback: decide what actually failed, allow one bounded retry when it has a reasonable chance of working, and then send the same logical request to a compatible model elsewhere. OpenAI might be followed by Anthropic and Gemini, but every attempt must share one deadline and one application request ID.

The API calls are the easy part. The difficult work sits between them: translating message formats, handling streams that fail after text has reached the user, preventing tools from running twice, interpreting ambiguous timeouts, and accounting for failed calls that still cost money.

What provider fallback is

Provider fallback gives one logical request an ordered set of provider routes. The application request ID, conversation state, required capabilities, and output contract stay fixed. Each physical call gets its own provider request ID.

Fallback sits among several reliability patterns, each operating at a different level. A same-provider retry changes only the attempt number and timing, which makes it useful for a brief transient failure. Switching models or deployments within one provider handles a model-specific limit or outage without changing the provider. Provider fallback crosses that boundary, usually changing both provider and model to survive provider-level or account-level failure.

Load balancing and cost-based routing happen before a failure. Load balancing distributes healthy traffic for capacity, latency, or cost management, while cost-based routing selects a provider and model specifically to optimize spend. Disaster recovery is wider in scope: it restores the application after a major incident rather than deciding where one model request should go. At the other end of the spectrum, returning a hardcoded response makes no model call at all. That can provide graceful degradation, but it is not equivalent generation.

An ordered route can place OpenAI first, Anthropic second, and Gemini third. That order records the application's operational preference. Model compatibility still has to be established separately through capability checks and evaluation.

Why production applications use it

Outages get the attention, but rate limits cause more day-to-day trouble. Provider accounts can cap requests and tokens, sometimes with a separate spend limit. Anthropic notes that a per-minute allowance may be enforced over shorter intervals, which means a burst can trigger 429s while the dashboard still looks comfortable. Gemini reports exhausted quota as 429 RESOURCE_EXHAUSTED. In both cases, the first response should be less pressure on the provider, not an unbounded retry loop. (Anthropic rate limits; Gemini troubleshooting)

Availability is also a product-level question. A provider may be up while its P99 latency is well beyond your response deadline. A region can degrade on its own. A model can disappear from an account, move behind a different entitlement, or become unavailable where your data must stay. From the user's perspective, each case has the same result: the application cannot answer in time.

Fallback does not solve every dependency failure. Bad requests and broken tools can take down every route; so can a shared network problem. It also introduces more cost and more variation between answers. For an internal tool or a non-urgent batch job, a clear error and a queue may be the more honest design. The extra provider is justified when its recovery value exceeds the ongoing cost of testing and operating another route.

Which failures should trigger fallback

Do not turn the table below into a switch statement keyed only by status code. A 429 with Retry-After: 1 and ten seconds left is workable. A 60-second delay with two seconds left is not. Likewise, a connection timeout and a timeout after the provider accepted the request carry different risks.

The official SDK defaults are a useful starting point, not a complete policy. OpenAI's Node SDK retries connection errors, 408, 409, 429, and 5xx responses by default. Anthropic's SDK retries connection errors, 429s, and 5xx responses, including its documented 529 overload status. Gemini documents separate guidance for 429, 500, 503, and 504. The router still has to enforce the deadline and count every physical call. (OpenAI Node retries; Anthropic errors; Gemini troubleshooting)

Designing the fallback chain

Start with a priority chain. It is predictable during an incident and easy to explain afterward:

1. OpenAI primary
2. Anthropic first fallback
3. Gemini second fallback

Keep provider model IDs in configuration. They change too often to deserve a source-code deployment, and a stable route name such as support-answer-v1 gives the application a contract that can be tested independently of those changes.

Priority should be applied only after eligibility. First remove routes that cannot satisfy the request's tools, media, context length, schema, or data-region rules. What remains can be ranked in a fixed order or by recent latency, with cost used as the main policy or a tiebreaker. Selecting the cheapest model before checking whether it can execute the tool schema optimizes the wrong part of the request.

Give the request an overall deadline as well as a timeout for each attempt. With a 12-second API contract, one reasonable budget is four seconds for the primary, three seconds for each fallback, and two seconds for backoff and response handling. Never begin a three-second call when only 400 milliseconds remain.

The route also needs a hard physical-attempt limit. Otherwise, two hidden SDK retries across three providers can become nine calls. Disable SDK retries when the router owns this policy, or include those calls in the same budget.

Concise routing flow

Validate request and required capabilities
  -> remove ineligible or open-circuit routes
  -> attempt primary within per-provider timeout
  -> success: validate and return
  -> non-recoverable failure: return typed error
  -> transient failure: retry if delay fits overall deadline
  -> still failing: choose next compatible healthy route
  -> no route or no time: return final typed error

Circuit breakers and health

Once a route is clearly unhealthy, new requests should stop proving the same point. Track circuit state by provider, model, region, and credential pool. Open the circuit after a failure threshold or error-rate window is crossed. After a cooldown, admit a small number of half-open probes; a successful probe closes the circuit, while a failed one opens it again. AWS's guidance describes the same two responsibilities: stop calls that are likely to fail, then probe carefully for recovery. (Circuit-breaker pattern)

Real traffic is the best health signal. Status pages can lag, and a synthetic prompt may succeed even when your account, model, or region is broken. Use passive error and latency windows, backed by sparse and inexpensive probes. A stream of paid “health-check prompts” is rarely justified.

Why models are not interchangeable

Moving from OpenAI to Anthropic takes more than swapping a model name. OpenAI exposes system and developer instruction roles in its APIs. Anthropic's Messages API keeps system instructions outside the messages array. Gemini's generateContent accepts systemInstruction and represents conversation turns with user and model roles. An adapter has to preserve the intent of the conversation without claiming that these instruction hierarchies are equivalent.

The same warning applies elsewhere:

• Tool definitions and tool-call IDs have different envelopes. Anthropic streams tool inputs as partial JSON deltas; Gemini returns native function-call parts; OpenAI emits its own function-call argument events or deltas.
• Structured output is not one universal JSON Schema implementation. Gemini supports a documented subset, while Claude documents its own limits and invalid-output conditions. Validate locally even when constrained decoding is enabled. (Gemini structured output; Claude structured output)
• Refusal and safety outcomes appear in different fields and follow different policies. Gemini can block prompts or candidates through safety feedback; other providers expose refusals or content-filter finish states differently.
• Context windows, tokenizers, token accounting, stop-sequence limits, temperature ranges, reasoning controls, and multimodal formats vary by model.
• Streaming protocols may all use SSE at the HTTP level while carrying unrelated event schemas.

Build capability-compatible fallback groups instead of one global provider list. Plain-text summarization may need only text input and output. A support agent might require strict tool schemas, parallel calls, a 100,000-token context window, streamed arguments, and an approved data region. A route belongs in that group only after it passes the same contract tests as the primary.

Passing the capability checks does not establish equivalent behavior. Evaluate answer quality and refusals alongside tool choice, schema validity, latency, and cost across the full route. When a model change could alter a consequential business decision, the route policy should require the approved model behavior and return an explicit failure if no qualifying provider is available.

A Node.js and TypeScript implementation

The example below is a working baseline for non-streaming text generation on Node.js 20+. It uses the official OpenAI and Anthropic SDKs and Gemini's REST generateContent endpoint. Gemini is called through fetch so every adapter obeys the same AbortSignal and the router owns the retry count. The request mapping is deliberately text-only; tools, media, and provider-native structured output need their own adapters.

Install the SDKs:

pnpm add openai @anthropic-ai/sdk

import OpenAI from "openai";
import Anthropic from "@anthropic-ai/sdk";
import { randomUUID } from "node:crypto";

type Provider = "openai" | "anthropic" | "gemini";
type Role = "system" | "user" | "assistant";

type Message = { role: Role; content: string };

type GenerateRequest = {
  requestId: string;          // Created by the application, not a provider
  messages: Message[];
  maxOutputTokens: number;
  temperature?: number;
};

type NormalizedResponse = {
  requestId: string;
  providerRequestId?: string;
  provider: Provider;
  model: string;
  text: string;
  finishReason?: string;
  usage: { inputTokens?: number; outputTokens?: number };
};

type AttemptContext = {
  signal: AbortSignal;
  model: string;
};

interface ProviderAdapter {
  readonly provider: Provider;
  generate(
    request: GenerateRequest,
    context: AttemptContext
  ): Promise<NormalizedResponse>;
}

type FailureKind =
  | "timeout"
  | "rate_limit"
  | "server"
  | "network"
  | "authentication"
  | "invalid_request"
  | "unknown";

class UpstreamError extends Error {
  constructor(
    message: string,
    readonly provider: Provider,
    readonly kind: FailureKind,
    readonly status?: number,
    readonly retryAfterMs?: number,
    readonly cause?: unknown
  ) {
    super(message);
    this.name = "UpstreamError";
  }
}

function retryAfterMs(headers: unknown): number | undefined {
  if (!headers || typeof headers !== "object") return undefined;
  const h = headers as { get?: (name: string) => string | null };
  const raw = h.get?.("retry-after");
  if (!raw) return undefined;
  const seconds = Number(raw);
  if (Number.isFinite(seconds)) return Math.max(0, seconds * 1_000);
  const date = Date.parse(raw);
  return Number.isNaN(date) ? undefined : Math.max(0, date - Date.now());
}

function classifyStatus(status?: number): FailureKind {
  if (status === 408 || status === 504) return "timeout";
  if (status === 429) return "rate_limit";
  if (status !== undefined && status >= 500) return "server";
  if (status === 401 || status === 402 || status === 403)
    return "authentication";
  if (status !== undefined && status >= 400) return "invalid_request";
  return "unknown";
}

function wrapError(provider: Provider, error: unknown): UpstreamError {
  if (error instanceof UpstreamError) return error;
  if (error instanceof DOMException && error.name === "AbortError") {
    return new UpstreamError("Provider attempt timed out", provider, "timeout", undefined, undefined, error);
  }

  const e = error as {
    message?: string;
    status?: number;
    headers?: unknown;
    code?: string;
  };
  const status = typeof e?.status === "number" ? e.status : undefined;
  const networkCodes = new Set(["ECONNRESET", "ECONNREFUSED", "ENOTFOUND", "EAI_AGAIN"]);
  const kind = networkCodes.has(e?.code ?? "") ? "network" : classifyStatus(status);
  return new UpstreamError(
    e?.message ?? "Unknown provider error",
    provider,
    kind,
    status,
    retryAfterMs(e?.headers),
    error
  );
}

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  maxRetries: 0, // This router owns retries.
});

const anthropic = new Anthropic({
  apiKey: process.env.ANTHROPIC_API_KEY,
  maxRetries: 0,
});

const openAIAdapter: ProviderAdapter = {
  provider: "openai",
  async generate(request, { signal, model }) {
    try {
      const response = await openai.chat.completions.create(
        {
          model,
          messages: request.messages,
          max_completion_tokens: request.maxOutputTokens,
          temperature: request.temperature,
        },
        { signal }
      );
      return {
        requestId: request.requestId,
        providerRequestId: response._request_id,
        provider: "openai",
        model: response.model,
        text: response.choices[0]?.message.content ?? "",
        finishReason: response.choices[0]?.finish_reason ?? undefined,
        usage: {
          inputTokens: response.usage?.prompt_tokens,
          outputTokens: response.usage?.completion_tokens,
        },
      };
    } catch (error) {
      throw wrapError("openai", error);
    }
  },
};

const anthropicAdapter: ProviderAdapter = {
  provider: "anthropic",
  async generate(request, { signal, model }) {
    const system = request.messages
      .filter((m) => m.role === "system")
      .map((m) => m.content)
      .join("\n\n");
    const messages = request.messages
      .filter((m) => m.role !== "system")
      .map((m) => ({
        role: m.role === "assistant" ? "assistant" as const : "user" as const,
        content: m.content,
      }));

    try {
      const response = await anthropic.messages.create(
        {
          model,
          system: system || undefined,
          messages,
          max_tokens: request.maxOutputTokens,
          temperature: request.temperature,
        },
        { signal }
      );
      return {
        requestId: request.requestId,
        providerRequestId: response.id,
        provider: "anthropic",
        model: response.model,
        text: response.content
          .filter((block) => block.type === "text")
          .map((block) => block.text)
          .join(""),
        finishReason: response.stop_reason ?? undefined,
        usage: {
          inputTokens: response.usage.input_tokens,
          outputTokens: response.usage.output_tokens,
        },
      };
    } catch (error) {
      throw wrapError("anthropic", error);
    }
  },
};

const geminiAdapter: ProviderAdapter = {
  provider: "gemini",
  async generate(request, { signal, model }) {
    const apiKey = process.env.GEMINI_API_KEY;
    if (!apiKey) throw new Error("GEMINI_API_KEY is not configured");

    const systemInstruction = request.messages
      .filter((m) => m.role === "system")
      .map((m) => m.content)
      .join("\n\n");
    const contents = request.messages
      .filter((m) => m.role !== "system")
      .map((m) => ({
        role: m.role === "assistant" ? "model" : "user",
        parts: [{ text: m.content }],
      }));

    try {
      const response = await fetch(
        `https://generativelanguage.googleapis.com/v1beta/models/${encodeURIComponent(model)}:generateContent`,
        {
          method: "POST",
          signal,
          headers: {
            "content-type": "application/json",
            "x-goog-api-key": apiKey,
          },
          body: JSON.stringify({
            systemInstruction: systemInstruction
              ? { parts: [{ text: systemInstruction }] }
              : undefined,
            contents,
            generationConfig: {
              maxOutputTokens: request.maxOutputTokens,
              temperature: request.temperature,
            },
          }),
        }
      );

      if (!response.ok) {
        const body = await response.text();
        throw new UpstreamError(
          `Gemini ${response.status}: ${body.slice(0, 500)}`,
          "gemini",
          classifyStatus(response.status),
          response.status,
          retryAfterMs(response.headers)
        );
      }

      const body = await response.json() as {
        responseId?: string;
        modelVersion?: string;
        candidates?: Array<{
          content?: { parts?: Array<{ text?: string }> };
          finishReason?: string;
        }>;
        usageMetadata?: {
          promptTokenCount?: number;
          candidatesTokenCount?: number;
        };
      };
      const candidate = body.candidates?.[0];
      return {
        requestId: request.requestId,
        providerRequestId: body.responseId,
        provider: "gemini",
        model: body.modelVersion ?? model,
        text: candidate?.content?.parts?.map((p) => p.text ?? "").join("") ?? "",
        finishReason: candidate?.finishReason,
        usage: {
          inputTokens: body.usageMetadata?.promptTokenCount,
          outputTokens: body.usageMetadata?.candidatesTokenCount,
        },
      };
    } catch (error) {
      throw wrapError("gemini", error);
    }
  },
};

type Route = {
  adapter: ProviderAdapter;
  model: string;
  timeoutMs: number;
};

const required = (name: string): string => {
  const value = process.env[name];
  if (!value) throw new Error(`${name} is required`);
  return value;
};

const routes: Route[] = [
  { adapter: openAIAdapter, model: required("OPENAI_MODEL"), timeoutMs: 4_000 },
  { adapter: anthropicAdapter, model: required("ANTHROPIC_MODEL"), timeoutMs: 3_000 },
  { adapter: geminiAdapter, model: required("GEMINI_MODEL"), timeoutMs: 3_000 },
];

const sleep = (ms: number, signal: AbortSignal) =>
  new Promise<void>((resolve, reject) => {
    const timer = setTimeout(resolve, ms);
    signal.addEventListener("abort", () => {
      clearTimeout(timer);
      reject(signal.reason);
    }, { once: true });
  });

const isTransient = (error: UpstreamError) =>
  ["timeout", "rate_limit", "server", "network"].includes(error.kind);

function backoffMs(error: UpstreamError, retryNumber: number): number {
  if (error.retryAfterMs !== undefined) return error.retryAfterMs;
  const cap = Math.min(2_000, 250 * 2 ** retryNumber);
  return Math.floor(Math.random() * cap); // Full jitter
}

export async function generateWithFallback(
  input: Omit<GenerateRequest, "requestId"> & { requestId?: string },
  options: { overallTimeoutMs?: number; maxAttempts?: number } = {}
): Promise<NormalizedResponse> {
  const request: GenerateRequest = {
    ...input,
    requestId: input.requestId ?? randomUUID(),
  };
  const overallTimeoutMs = options.overallTimeoutMs ?? 12_000;
  const maxAttempts = options.maxAttempts ?? 4;
  const overall = new AbortController();
  const overallTimer = setTimeout(
    () => overall.abort(new DOMException("Overall deadline exceeded", "AbortError")),
    overallTimeoutMs
  );
  const deadline = Date.now() + overallTimeoutMs;
  let attempts = 0;
  let lastError: UpstreamError | undefined;

  try {
    for (const route of routes) {
      // At most one same-provider retry, and never exceed the global cap.
      for (let retry = 0; retry < 2 && attempts < maxAttempts; retry++) {
        if (overall.signal.aborted) throw overall.signal.reason;
        attempts += 1;

        const remainingMs = deadline - Date.now();
        if (remainingMs <= 0) {
          throw new DOMException("Overall deadline exceeded", "AbortError");
        }
        const attempt = new AbortController();
        const abortAttempt = () => attempt.abort(overall.signal.reason);
        overall.signal.addEventListener("abort", abortAttempt, { once: true });
        const attemptTimer = setTimeout(
          () => attempt.abort(new DOMException("Provider timeout", "AbortError")),
          Math.min(route.timeoutMs, remainingMs)
        );

        try {
          const result = await route.adapter.generate(request, {
            signal: attempt.signal,
            model: route.model,
          });
          // Record attempt/provider/model/latency here before returning.
          return result;
        } catch (error) {
          lastError = wrapError(route.adapter.provider, error);
          // Auth, invalid input, policy, and other non-transient failures stop.
          if (!isTransient(lastError)) throw lastError;

          const delay = backoffMs(lastError, retry);
          const canRetrySameProvider = retry === 0 && attempts < maxAttempts;
          if (!canRetrySameProvider || Date.now() + delay >= deadline) break;
          await sleep(delay, overall.signal);
        } finally {
          clearTimeout(attemptTimer);
          overall.signal.removeEventListener("abort", abortAttempt);
        }
      }
    }
    throw lastError ?? new Error("No provider route was attempted");
  } finally {
    clearTimeout(overallTimer);
  }
}

The shared request keeps one requestId, while each adapter records its provider request ID. Disabling SDK retries makes maxAttempts an honest count of physical calls. Each attempt can be cancelled, no attempt can outlive the overall deadline, and only timeout, rate-limit, server, and network failures move through the chain.

Authentication and invalid-request errors stop immediately. A team may choose to let a credential failure fall through to a separately billed provider, but that policy needs a loud alert and an open circuit. Otherwise, an expired primary key can stay hidden while the fallback quietly absorbs all production traffic and cost.

A production deployment needs more than this baseline: shared circuit state and capability filters, structured attempt logs, stream adapters, provider-specific error parsing, and the application's real schema and tool mappings. The installed SDK versions should be pinned and tested as part of that work, particularly their behavior when a streamed response body times out.

Retrying before fallback

A short retry is worthwhile when the problem may clear faster than a provider switch: a reset connection, a brief 500, or a 429 with a small Retry-After. Malformed requests and invalid keys should fail immediately. If the provider's requested delay does not fit inside the deadline, waiting is no longer a retry strategy; fall back or return an error.

Exponential backoff increases the delay ceiling after each failure. Full jitter chooses a random delay below that ceiling so thousands of workers do not wake together. AWS recommends this pattern for transient throttling, connectivity, and availability failures, with the important warning that excessive retries add contention to an already unhealthy service. (Retry with backoff)

A 12-second example might look like this:

Do not carry one provider's backoff delay over to an independent provider unless a client-wide concurrency limit requires it. The delay belongs to the resource that failed. For interactive traffic, keep the retry allowance small and let the overall deadline govern each decision. This budget must include calls initiated by SDKs and gateways as well as retries from API handlers, job runners, or browsers.

Streaming, tools, and structured output

Streaming changes the commitment point

Fallback is straightforward only until output reaches the client. Before the first token is committed, the gateway can discard an attempt and choose another route. After that point, switching models can repeat a sentence, contradict text already shown, lose tool state, or introduce a different safety decision.

The event protocols differ as well. Anthropic documents message_start, content-block events, deltas, and message_stop; tool inputs arrive as partial JSON, and an error event may follow an HTTP 200 response. Gemini emits GenerateContentResponse chunks. OpenAI uses its own typed events or chat-completion chunks. A normalized client stream must preserve the provider's stop and error details instead of flattening them away. (Anthropic streaming; Gemini streaming reference)

Pick a commitment policy and document it in the client contract:

• Fallback only before first meaningful output. Lowest latency and simplest contract. After output starts, terminate with a typed stream_error event and mark the response partial.
• Buffer a prefix. Hold the first complete sentence, N tokens, or a short time window. This gives the router a chance to fail over before committing, but worsens time to first token.
• Buffer the entire answer. Maximum fallback flexibility, but no longer a real user-visible stream.
• Application-specific continuation. A new request asks a model to continue from captured text. This is not transparent fallback and must be labeled and tested; tools and reasoning blocks make it unsafe.

Never splice two providers' output into what appears to be one uninterrupted answer. The client event envelope should distinguish started, delta, completed, and failed_after_partial_output. Include a retry-safety field so the application can decide whether to present a retry action after a partial response.

Tool calls require exactly-once effects, not exactly-once inference

Consider a call to create_support_ticket. The first provider returns valid arguments, the application creates ticket T-123, and the connection drops before the model receives the result. If the fallback sees only the original conversation, it may ask to create the same ticket again.

Normalize provider tool calls into a shape such as {callId, name, arguments}, then validate arguments against the local schema. Provider-side constraints are useful, but local validation remains the final authority. A streamed call must not execute until all arguments have arrived and passed validation.

For a side-effecting tool, derive an idempotency key from stable application state, for example:

tool:create_support_ticket:{applicationRequestId}:{logicalToolStep}

Store started, succeeded, or failed alongside the tool result. Repeating a successful key returns that stored result instead of creating another ticket. For an operation such as charge_account, the downstream payment call must receive its own idempotency key; a provider-specific tool-call ID is not enough.

Once a tool succeeds, write the normalized call and result into the canonical conversation before another model attempt. Never restart from the user's original message and replay completed work. A provider that cannot represent the current tool state faithfully is no longer eligible at that point in the agent loop.

Structured output is an application contract

Constrained generation reduces malformed JSON; it does not remove the need for local validation. Providers implement different schema subsets and stop conditions. Parse every response against the same versioned application schema, then classify the outcome:

1. Transport/provider failure: normal retry/fallback policy.
2. Truncated output, such as a token limit: retry with a safe larger limit if allowed.
3. Valid JSON that violates schema: one constrained repair or regeneration.
4. Semantically invalid data: application validation failure, not a provider outage.

Malformed output may fall back only to a route that supports the same schema features. Record this reason separately from transport errors, since the provider did return a response and the failure occurred during validation against the application contract.

Preventing duplicate work and charges

Request idempotency, response caching, provider retries, and application retries solve related but separate problems. Idempotency decides whether the application has already accepted a logical request. Create that request ID at the edge and persist it before contacting a provider. If the client submits the same key again, attach it to the operation already in progress or return the stored result.

A response cache asks a narrower question: whether these inputs and this policy already have a completed answer that can be reused. Its key may include normalized messages, route version, tools, schema, and sampling settings. Identical prompts can still represent distinct business operations, which is why a cache hit cannot replace the idempotency check.

Retries operate at two different boundaries. A provider retry sends another physical call to the same provider, usually after a transient error. An application retry occurs when the client or job submits the logical operation again. The application request ID connects those layers, preventing a client retry from being mistaken for new work while still allowing the router to account for each provider attempt.

Ambiguous timeouts are the awkward case. The client has stopped waiting, but the provider may continue generating. Falling back can therefore create a second bill. Cancellation is best effort and cannot prove that upstream work stopped. A small amount of duplicate inference may be unavoidable for plain generation, so measure it. For side effects, enforce idempotency in the tool or downstream API rather than trusting model cancellation.

Because upstream cancellation is uncertain, the system cannot credibly guarantee exactly-once LLM execution. It can still return one logical result when possible, use idempotency to provide at-most-once business side effects, and retain a complete record of each physical attempt for billing and incident analysis.

Observability and testing

Record attempts, not only final requests

A final 200 may hide two failed providers, several seconds of delay, and three billable calls. Emit one event for each attempt and a summary for the logical request. The attempt record should include:

• application request ID and attempt number
• provider, configured model, and returned model/version
• provider request ID
• start/end time and time to first token
• status and normalized error classification
• input/output tokens and estimated cost using a versioned price table
• retry or fallback reason
• final provider
• cache hit, miss, or bypass
• circuit state and route/region
• whether any output or tool side effect had already been committed

{
  "event": "llm_attempt_finished",
  "applicationRequestId": "req_app_8f31",
  "attempt": 2,
  "provider": "anthropic",
  "model": "support-answer-v1:anthropic",
  "providerRequestId": "req_provider_...",
  "startedAt": "2026-07-12T09:30:04.812Z",
  "durationMs": 2260,
  "timeToFirstTokenMs": 610,
  "status": "success",
  "errorClass": null,
  "inputTokens": 1840,
  "outputTokens": 322,
  "estimatedCostUsd": 0.0123,
  "fallbackReason": "openai_rate_limit",
  "finalProvider": "anthropic",
  "cacheStatus": "miss",
  "circuitState": "closed"
}

The cost above illustrates the log shape rather than a current price. Calculate it from a versioned price table so historical records do not change when a provider updates its pricing.

The operational dashboard needs the primary failure rate alongside the share recovered by fallback. It should also show the latency and cost added during recovery, plus the number of open and half-open routes. Track these measurements by route and failure reason, including P50/P95/P99 latency and time to first token. Alert on sharp fallback-rate changes, sustained recovery failure, primary authentication errors, unusually expensive discarded attempts, and circuits that remain open longer than expected.

Test the failure states deliberately

Do not use a real outage as the first test of the fallback path. Adapter fakes and a local fault-injection proxy cover most router behavior, while a non-production provider project reveals SDK and protocol differences that mocks can miss.

Begin with throttling and timeouts. Inject a 429 with both short and long Retry-After values: the short delay should retry, whereas the long delay should fall back when the deadline requires it. Stall response headers, then stall the body. In both timeout cases, the attempt must abort, the overall deadline must take precedence, and the acceptance ambiguity must be logged. A 502/503/529 sequence exercises a broader provider outage; once its threshold is reached, the circuit should open and later traffic should skip that route.

Output failures need separate tests because the transport may have succeeded. Return invalid JSON or data that violates the schema, then assert that neither the tool nor the output is consumed and that the router performs only a bounded repair or moves to a compatible fallback. For streaming, disconnect once before the first token and once after it. The first case should fall back cleanly; the second should produce a typed partial failure. When every route returns a transient error, verify a stable final error and the exact maximum attempt count.

Side effects and recovery state deserve their own scenarios. Lose the response after a tool has changed business state, repeat the call, and confirm that the same idempotency key returns the stored result. Advance the circuit cooldown and test both a successful and a failed probe, including half-open concurrency and the resulting state transition. A 401 or 400 should produce an immediate typed failure and alert without a retry storm. Finally, slow every attempt until the deadline is nearly exhausted and confirm that the router refuses to start another route without enough time remaining.

Contract tests should also cover conversation translation, stop reasons, replayed tool results, supported schema features, and safety outcomes. Token accounting and model deprecation checks belong in the same suite. Run these cases continuously so changes to an SDK or provider do not leave a rarely used route broken until the next incident.

Build it or use a gateway

An in-service implementation is reasonable for one text-only workload, unusual routing requirements, or a team that considers this infrastructure strategically important. It avoids another network hop and keeps policy ownership local. The cost is ongoing ownership of API drift, credentials, retry accounting, stream normalization, dashboards, circuit state, and on-call failures.

A gateway starts to earn its place when several applications need the same interface, provider credentials, fallback rules, analytics, cost attribution, cache, and rate limits. Mature designs keep retries, fallback selection, health/cooldown, and routing strategy as separate controls. Preserve that separation whether the code lives in a gateway or an application. (LiteLLM routing; Kong retry and fallback)

The gateway itself becomes shared infrastructure. Deploy it across failure domains, give it explicit deadlines, protect its state stores, and define how applications behave when it is unavailable.

Implementing fallback with Nirmos

Nirmos can hold the policy between application code and provider APIs. The application sends an OpenAI-compatible request, the gateway resolves a route alias, adapters translate it, and the route applies timeouts, retry limits, fallback eligibility, and attempt logging. Provider keys remain in central credential management instead of browser or application deployments. Request metadata supplies the application, user, conversation, environment, and tags needed for attribution.

Conceptually, a route might declare:

route: support-answer-v1
overall_timeout_ms: 12000
max_attempts: 4
providers:
  - provider: openai
    model: ${OPENAI_MODEL}
    timeout_ms: 4000
  - provider: anthropic
    model: ${ANTHROPIC_MODEL}
    timeout_ms: 3000
  - provider: gemini
    model: ${GEMINI_MODEL}
    timeout_ms: 3000
fallback_on: [timeout, rate_limit, provider_5xx]
do_not_fallback_on: [invalid_request, authentication, policy_rejection]

For an existing server-side OpenAI SDK integration, the migration should touch only the credential, base URL, and model or route identifier:

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.NIRMOS_API_KEY,
  baseURL: process.env.NIRMOS_BASE_URL,
});

const response = await client.chat.completions.create({
  model: process.env.NIRMOS_ROUTE_ID!, // e.g. a tested fallback route alias
  messages: [{ role: "user", content: "Summarize this incident report." }],
});

Confirm the endpoint, headers, route schema, provider support, caching, rate-limit controls, log fields, and cost calculation against the current Nirmos release. Planned capabilities should be labelled as such. The application should continue to send one logical request ID, while the configured route bounds physical attempts and filters providers by capability. Attempt logs should record the reason whenever the router moves to another provider.

Production checklist

• Classify provider, network, validation, policy, and application failures separately.
• Retry only transient failures and honor Retry-After when it fits the deadline.
• Set per-provider timeouts, one overall deadline, and a hard physical-attempt cap.
• Disable or budget SDK, gateway, job, and client retries together.
• Define and contract-test capability-compatible model groups.
• Decide what happens before first token and after partial stream output.
• Validate structured output locally against a versioned schema.
• Normalize tool calls and make side-effecting tools idempotent.
• Preserve canonical conversation and completed tool results across attempts.
• Log every physical attempt under one application request ID.
• Monitor fallback rate, tail latency, recovery rate, and wasted-attempt cost.
• Scope circuit breakers by provider/model/region/credential and test recovery.
• Inject 429s, timeouts, malformed output, stream breaks, and total failure.
• Return user-facing errors that distinguish retryable, partial, and final failure.
• Review provider models, limits, SDK behavior, pricing, and route compatibility regularly.

Before moving to another route, the router has to classify whether the failure is temporary and verify that the candidate can satisfy the request. It must also compare the candidate's timeout with the time left in the overall deadline. These decisions happen under one application operation, with each physical attempt recorded separately and each side effect protected by its own idempotency control.

FAQ

How do I automatically fall back from OpenAI to Anthropic?

Put both APIs behind adapters, classify the OpenAI failure, and call Anthropic only for eligible transient cases such as a bounded 429, timeout, network error, or provider 5xx. Translate system instructions, conversation roles, tools, and output schemas explicitly. Keep one application request ID and record a separate request ID for each provider attempt.

Should an OpenAI 429 always trigger fallback?

Check Retry-After against the remaining deadline first. A short wait leaves room for one same-provider retry, which may be cheaper and more consistent than changing models. If the delay does not fit, the limit appears sustained, or that retry fails, continue to an eligible fallback. Jitter keeps workers from retrying in lockstep.

Can I switch from OpenAI to Gemini without changing the prompt?

A plain text prompt may carry over unchanged, but a production request still needs translation and evaluation. Instruction roles, tool envelopes, safety outcomes, schema subsets, stop reasons, context limits, and streaming events differ. Treat the canonical request as provider-neutral intent that each adapter compiles into the provider's native format.

Should content-policy rejection trigger another provider?

A content-policy rejection should end the generic outage-fallback path. Return the product's refusal or error response rather than trying providers until one accepts the prompt. When an application has a legitimate reason to account for differences between provider policies, encode that behavior in an explicit, reviewed routing policy.

Can fallback continue a response after streaming has started?

Another model can be prompted with the partial text, but that is a new continuation request rather than transparent fallback. It may repeat or contradict the answer, and partial tool or reasoning blocks cannot be transferred safely. Most systems should switch only before the first committed output and return a typed partial-stream failure afterward.

How many retries and fallback attempts should I allow?

There is no universal number. For interactive traffic, start with one short retry and one or two alternative routes under a single deadline. Hidden SDK retries count as physical attempts. Expand the budget only when measured recovery rate justifies the added latency and cost.

Does idempotency prevent duplicate LLM charges?

Not necessarily. Idempotency prevents the application from accepting one logical operation twice and protects side-effecting tools. A provider may continue processing after the client times out, so fallback can still create a second inference charge. Record ambiguous attempts and enforce idempotency wherever business state changes.

Is an AI gateway required for multi-provider fallback?

One application can own its adapters and routing directly. The gateway becomes useful later, when several services need to share credentials and route configuration along with logs, rate limits, caching, cost attribution, and consistent fallback behavior.