Streaming LLM Responses to UI

A non-streamed LLM call feels broken before it feels slow: the user sends a request and stares at a blank state for anywhere from two to twenty seconds while the model generates the full response server-side. Streaming turns that dead air into a typing effect — tokens appear as the model produces them, so perceived latency drops even though total generation time is identical. For anything chat-shaped, or any UI where an LLM is drafting text a human will read live, streaming isn't a nice-to-have, it's the difference between a tool that feels responsive and one people stop using.

Why streaming changes perceived latency

Time-to-first-token (TTFT) is what users actually judge, not total completion time. A response that takes 8 seconds end-to-end but starts rendering at 400ms reads as fast; the same 8-second response delivered as one blob at second 8 reads as hung. This matters more with larger models and longer outputs — a 1,500-token answer from a frontier model can take several seconds to generate in full, and non-streaming forces the whole wait up front with zero feedback that anything is happening.

There's a secondary benefit specific to agentic or tool-using flows: streaming lets you show intermediate state (“searching records…”, “calling create_invoice…”) instead of a single opaque spinner covering a multi-step chain.

SSE vs WebSockets vs plain chunked HTTP

For LLM output specifically, the traffic is one-directional (server to client) after the initial request, so a full-duplex protocol like WebSockets is usually more infrastructure than the problem needs. Server-Sent Events (SSE) is the pragmatic default: it rides over plain HTTP, works through most corporate proxies and load balancers without special configuration, reconnects automatically via EventSource, and both the OpenAI and Anthropic APIs emit their streaming responses as SSE natively — so a browser client can, in the simplest case, forward the same event format the model API sent.

WebSockets earn their cost when you need bidirectional interaction mid-generation — the user is expected to interrupt, steer, or send follow-ups while tokens are still arriving, or you're multiplexing many independent streams over one connection. For a single request/response chat turn, that's usually unneeded complexity. Chunked transfer-encoding without SSE framing works too, but you lose the built-in event typing and reconnection semantics for very little benefit over SSE.

Proxying the stream through your own backend

You rarely want the browser calling the model API directly — that leaks your API key client-side. The usual shape is: browser opens a fetch/EventSource connection to your backend, your backend opens its own streaming request to Claude or OpenAI, and re-emits each delta to the browser as it arrives, adding your own SSE framing on top if you need custom event types (e.g. a distinct event for tool calls vs. text).

app.post('/api/chat', async (req, res) => {
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Connection', 'keep-alive');

  const stream = await anthropic.messages.stream({
    model: 'claude-sonnet-4-5',
    max_tokens: 1024,
    messages: req.body.messages,
  });

  req.on('close', () => stream.controller.abort()); // client cancelled

  for await (const event of stream) {
    if (event.type === 'content_block_delta' && event.delta.type === 'text_delta') {
      res.write(`data: ${JSON.stringify({ text: event.delta.text })}\n\n`);
    }
  }
  res.write('data: [DONE]\n\n');
  res.end();
});

The req.on('close', ...) line matters as much as the happy path: if the user navigates away or cancels, propagate that abort down to the upstream model call. Otherwise you keep paying for tokens nobody will read, and under load that's a real cost line, not a theoretical one.

Rendering partial text and partial structured output

Plain prose is easy — append each delta to a buffer and re-render. Markdown is slightly harder: an unclosed code fence or bold marker mid-stream will render wrong for a frame or two. Most chat UIs accept this and let the markdown parser re-run on each chunk (it self-corrects once the closing token arrives); trying to fully suppress the flicker is rarely worth the complexity.

Structured output (JSON mode / tool calls) is the harder case, because the payload isn't valid JSON until the last token lands. Don't try to JSON.parse a growing string on every delta — it will throw on every intermediate chunk. Either wait for the block to close before parsing (fine if the JSON is small and arrives quickly), or use a streaming/partial JSON parser that tolerates truncated input and exposes whatever keys have resolved so far. If you're rendering a tool-call argument live — say, showing a search query as it's being typed by the model — a partial-JSON tolerant parser is the only way to do that without flashing parse errors.

Backpressure, error handling, and cancellation mid-stream

A stream can fail after it's already started — rate limits, a dropped upstream connection, a content filter trip. Because headers are already sent (200 OK, text/event-stream), you can't fall back to a normal HTTP error response once generation begins; you have to send an error as an SSE event and let the client interpret it. Design one dedicated event type for this from the start (an event: error line or a JSON payload with a type: "error" field) rather than discovering mid-incident that your client has no way to distinguish a truncated-but-successful stream from one that died.

Cancellation should be cheap and explicit in both directions: the client aborts its fetch/EventSource (call .close() or abort the AbortController) when the user clicks stop or navigates away, and your server must be wired to notice that disconnect and abort the upstream model call — see the req.on('close') handler above. Skipping that half of the loop is the single most common streaming bug: the UI looks cancelled but the backend keeps burning tokens against the model API in the background.

UI details that matter more than they seem

A few small things separate a streaming UI that feels solid from one that feels janky: batch DOM updates (append to a string buffer and flush on requestAnimationFrame rather than re-rendering on every single delta, especially for React where naive per-token state updates will visibly stutter), show a distinct "thinking" state before the first token arrives versus mid-stream, auto-scroll only if the user hasn't manually scrolled up, and disable the input/send button until the stream closes or errors so a second request can't race the first.

None of this requires a framework. The core loop — open a stream, forward deltas, tolerate partial JSON, propagate cancellation both ways — is a few dozen lines on top of whatever HTTP server you already run, and it's the difference users actually notice between a demo and a product.