Docs: streamText timeout/abortSignal deadlines discard fallback output (generateText recovers)

[zirkelc]

Summary

When the AI SDK aborts a call via a streamText-level deadline — timeout.stepMs, timeout.chunkMs, timeout.totalMs, or a caller-supplied abortSignal — an ai-retry fallback cannot recover the request, even though the retry is correctly attempted. streamText tears the stream down and discards whatever the fallback produces.

The same deadlines on generateText do recover. So this is specific to streamText, and it's a sharp edge worth documenting (with guidance on where to put deadlines so fallbacks actually work).

This assumes the AI SDK tags step/chunk timeout aborts as TimeoutError (native in ai v7, see vercel/ai#14943; backportable to v6). Without that tagging, stepMs/chunkMs abort with a generic AbortError and ai-retry doesn't even attempt the fallback (resolveAbortSignal keeps the spent signal). With it, the fallback is attempted — and this issue is about what happens next.

Behavior matrix

Caller	Deadline	Fallback recovers?
streamText	timeout.chunkMs	❌
streamText	timeout.stepMs	❌
streamText	timeout.totalMs	❌
streamText	inbound abortSignal: AbortSignal.timeout()	❌
generateText	timeout.stepMs	✅
generateText	timeout.totalMs	✅
either	model-layer error (doStream/doGenerate throws)	✅
either	ai-retry per-attempt timeout	✅

Root cause

streamText builds a single master signal:

abortSignal: mergeAbortSignals(
  abortSignal,                                  // caller-supplied
  totalTimeoutMs != null ? AbortSignal.timeout(totalTimeoutMs) : undefined,
  stepAbortController?.signal,
  chunkAbortController?.signal,
)

Its streaming pipeline independently watches that master signal and emits an abort stream part / sets isAborted the moment it fires — regardless of the model layer. ai-retry sits below streamText (it wraps model.doStream): it catches the primary's abort and runs the fallback on a fresh signal, but the fallback's stream arrives into a pipeline that has already finalized as aborted, so the output is discarded.

generateText has no such pipeline — it just await model.doGenerate({ abortSignal }). ai-retry intercepts below, the fallback returns, and generateText returns that result. No separate teardown throws it away.

Implication: the only deadlines an ai-retry fallback can survive under streamText are ones enforced below streamText — i.e. ai-retry's own per-attempt timeout (which aborts the attempt's own signal, not streamText's master signal), or a stall-aware wrapper inside the model layer. A deadline set on streamText itself is, by construction, unrecoverable.

Runnable repro

/**
 * Which AI SDK "deadline" mechanisms allow an ai-retry fallback to recover.
 * Primary always stalls (errors when its signal aborts); fallback always returns "FALLBACK_OK".
 * Requires step/chunk timeouts tagged as TimeoutError (ai v7, or vercel/ai#14943 backported to v6).
 *
 * Run: pnpm tsx repro.mjs
 */
import { generateText, streamText } from 'ai';
import { MockLanguageModelV3 } from 'ai/test';
import { createRetryable } from 'ai-retry';

const TEXT = 'FALLBACK_OK';

/** AbortSignal.timeout() uses an unref'd timer; keep the loop alive in this isolated repro. */
const keepAlive = setInterval(() => {}, 50);

const streamPrimary = new MockLanguageModelV3({
  modelId: 'primary-stall',
  doStream: async ({ abortSignal }) => ({
    stream: new ReadableStream({
      start(c) {
        c.enqueue({ type: 'stream-start', warnings: [] });
        if (abortSignal?.aborted) c.error(abortSignal.reason);
        else abortSignal?.addEventListener('abort', () => c.error(abortSignal.reason), { once: true });
      },
    }),
    rawCall: { rawPrompt: null, rawSettings: {} },
  }),
});

const streamFallback = new MockLanguageModelV3({
  modelId: 'fallback-ok',
  doStream: async () => ({
    stream: new ReadableStream({
      start(c) {
        c.enqueue({ type: 'stream-start', warnings: [] });
        c.enqueue({ type: 'text-start', id: '1' });
        c.enqueue({ type: 'text-delta', id: '1', delta: TEXT });
        c.enqueue({ type: 'text-end', id: '1' });
        c.enqueue({ type: 'finish', finishReason: 'stop', usage: { inputTokens: 1, outputTokens: 1, totalTokens: 2 } });
        c.close();
      },
    }),
    rawCall: { rawPrompt: null, rawSettings: {} },
  }),
});

const genPrimary = new MockLanguageModelV3({
  modelId: 'primary-stall',
  doGenerate: async ({ abortSignal }) =>
    new Promise((_res, rej) => {
      if (abortSignal?.aborted) rej(abortSignal.reason);
      else abortSignal?.addEventListener('abort', () => rej(abortSignal.reason), { once: true });
    }),
});

const genFallback = new MockLanguageModelV3({
  modelId: 'fallback-ok',
  doGenerate: async () => ({
    content: [{ type: 'text', text: TEXT }],
    finishReason: 'stop',
    usage: { inputTokens: 1, outputTokens: 1, totalTokens: 2 },
    warnings: [],
  }),
});

const retryable = (primary, fallback) =>
  createRetryable({ model: primary, retries: [{ model: fallback, timeout: 90_000 }] });

async function runStream(label, opts) {
  const result = streamText({ model: retryable(streamPrimary, streamFallback), prompt: 'hi', maxRetries: 0, onError: () => {}, ...opts });
  let text = '';
  const drain = (async () => {
    for await (const part of result.fullStream) if (part.type === 'text-delta') text += part.text ?? part.delta ?? '';
  })();
  await Promise.race([drain.catch(() => {}), new Promise((r) => setTimeout(r, 3000))]);
  console.log(`${label.padEnd(34)} recovered=${text.includes(TEXT)}`);
}

async function runGenerate(label, opts) {
  let text = '';
  try {
    const res = await generateText({ model: retryable(genPrimary, genFallback), prompt: 'hi', maxRetries: 0, ...opts });
    text = res.text;
  } catch {}
  console.log(`${label.padEnd(34)} recovered=${text.includes(TEXT)}`);
}

await runStream('streamText  timeout.chunkMs', { timeout: { chunkMs: 300 } });
await runStream('streamText  timeout.stepMs', { timeout: { stepMs: 300 } });
await runStream('streamText  timeout.totalMs', { timeout: { totalMs: 300 } });
await runStream('streamText  abortSignal.timeout', { abortSignal: AbortSignal.timeout(300) });
await runGenerate('generateText timeout.stepMs', { timeout: { stepMs: 300 } });
await runGenerate('generateText timeout.totalMs', { timeout: { totalMs: 300 } });

clearInterval(keepAlive);
process.exit(0);

Output

streamText  timeout.chunkMs        recovered=false
streamText  timeout.stepMs         recovered=false
streamText  timeout.totalMs        recovered=false
streamText  abortSignal.timeout    recovered=false
generateText timeout.stepMs        recovered=true
generateText timeout.totalMs       recovered=true

Versions

• ai-retry@1.9.0
• ai@6.0.193 (patched to tag step/chunk timeouts as TimeoutError; equivalent to ai v7 native behavior — fix(ai): tag step/chunk timeout aborts with TimeoutError reason vercel/ai#14943)

Suggested docs addition

A "Timeouts & fallback recovery" note covering:

1. Deadlines set on streamText (timeout.* or abortSignal) are unrecoverable. Once the merged signal aborts, streamText finalizes the stream as aborted and discards any fallback ai-retry produces underneath. generateText does not have this limitation.
2. To make a stalled/slow attempt fall back under streamText, enforce the deadline at the model layer — use ai-retry's per-attempt timeout on each retry entry (it aborts the attempt's own signal, not streamText's master signal). streamText's timeout.totalMs is fine as a hard overall ceiling, but understand it cannot fall back.
3. Note that ai-retry's per-attempt timeout is a total per-attempt deadline (not reset per chunk), so it does not replicate streamText's chunkMs inter-chunk idle detection. Recovering from a mid-stream stall (provider sends headers, then goes silent) while still falling back would need a stall-aware idle timeout inside the model wrapper — possible future feature.

Suggested direction: make the streamText call itself retryable

Following from the root cause above: a deadline set on streamText (timeout.chunkMs / timeout.stepMs / timeout.totalMs, or an inbound abortSignal) can only be recovered by re-running the whole streamText call, so the next attempt gets a fresh master signal. The recovery has to happen above streamText — a model-layer retryable (today's createRetryable, which wraps doStream) structurally cannot reach it, because by the time the deadline fires the call it lives inside is already torn down.

It would be valuable for ai-retry to additionally offer a way to make the streamText call itself retryable — wrapping the call and re-running it with the next model when an attempt ends, before producing any output, due to a recoverable deadline. That is the one place a chunkMs/stepMs/totalMs stall can actually fail over.

This complements the existing model-layer retry rather than replacing it:

• Provider errors and content-filter switches still recover best at the model layer — transparent, no re-run, no buffering (and the mid-stream-before-content fix from Mid-stream retry before first content loses fallback output (duplicate stream-start) #51).
• Only the streamText-level deadlines need the outer wrapper, and only before content has been emitted (after that, re-running would duplicate output — so the wrapper would gate on "nothing streamed yet", the same boundary documented in Streaming: Document retry limitation after first chunk #33).

(Leaving the API shape to you — this is just the direction.)

---

Note: #54 reported the same root cause scoped to chunkMs/stepMs and proposed this wrapping direction; folding it in here since this issue already covers all four deadline mechanisms.