Structured Output from LLMs — A Field Guide

Asking an LLM to "reply in JSON" and hoping for the best works in a demo and breaks in production. The model wraps the object in a markdown fence, adds a sentence of preamble, drops a trailing comma, or quietly renames a field. None of that shows up until a downstream parser throws on request number 4,000. Getting structured output that a program can trust means treating the schema as a contract enforced at the API level, not as a suggestion in the prompt.

Why prompted JSON fails

A plain instruction like "respond only with JSON matching this shape" relies on the model's training to produce well-formed output, and it usually does — until the input is unusual, the schema is deep, or the model decides an explanation is helpful and prepends one anyway. The common failure modes are markdown code fences around the object, single quotes instead of double, trailing commas, numbers rendered as strings ("42" instead of 42), and enums that drift ("in_progress" vs "In Progress" vs "inprogress"). Every one of these is a silent break for a naive json.loads call, and every one of them is more likely under load, not less — the failure rate does not average out, it clusters on the inputs your prompt didn't anticipate.

The fix is not a better prompt. It's using the provider's constrained decoding path instead of asking nicely.

JSON mode vs. tool-use-as-structured-output

Most providers now expose two different mechanisms, and they are not interchangeable. JSON mode (OpenAI's response_format: {"type": "json_object"}, or the stricter json_schema variant) constrains decoding so the model can only emit tokens that form valid JSON, optionally validated against a schema at generation time. It's the right tool when the output is genuinely "fill in this document" — a single object, no branching logic. Tool-use / function-calling (Anthropic's tools parameter, OpenAI's tools with tool_choice) asks the model to call a named function with arguments matching a JSON Schema. This is the better fit when the "output" is really a decision among several possible actions, or when you want the model to choose between multiple schemas — one tool per intent.

In practice, forcing a single tool call (tool_choice: {"type": "tool", "name": "extract_invoice"} on Anthropic, or tool_choice: {"type": "function", "function": {"name": "..."}} on OpenAI) is the most reliable structured-output primitive available today, because it constrains generation to the tool's declared schema rather than relying on the model choosing to follow format instructions in free text.

{
  "name": "extract_invoice",
  "description": "Extract structured invoice data from raw text",
  "input_schema": {
    "type": "object",
    "properties": {
      "invoice_number": { "type": "string" },
      "total_amount": { "type": "number" },
      "currency": { "type": "string", "enum": ["USD", "KES", "EUR"] },
      "line_items": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "description": { "type": "string" },
            "quantity": { "type": "integer" },
            "unit_price": { "type": "number" }
          },
          "required": ["description", "quantity", "unit_price"]
        }
      }
    },
    "required": ["invoice_number", "total_amount", "currency", "line_items"]
  }
}

Schema design: flat beats clever

The schema you hand the model is not just documentation — it's part of the prompt, and it shapes what the model produces. Deeply nested objects, heavy use of oneOf/anyOf, and fields marked optional "just in case" all increase the chance of a malformed or incomplete response. A few rules that hold up in practice: keep nesting to two or three levels; make fields required unless there's a genuine reason a value won't exist, because optional fields get skipped under ambiguity; use enums instead of free-text strings for anything with a closed set of values, since an enum constrains generation and a free-text field invites synonyms; and avoid additionalProperties: true — an open schema is an invitation for the model to invent keys you didn't ask for.

Validate, don't trust

Constrained decoding reduces malformed output; it does not eliminate semantically wrong output. A tool call can be syntactically perfect JSON and still put a negative number in quantity, an out-of-range date, or a currency code the enum forgot to include. Parse the response through a real schema validator — Pydantic in Python, Zod in TypeScript — rather than trusting the shape because it came from a "structured output" call. This is also where you catch the provider-side edge case where JSON mode is enabled but the model still returns an empty object or an object missing a required key, which does happen, particularly on longer contexts or unusual inputs.

from pydantic import BaseModel, Field, ValidationError

class LineItem(BaseModel):
    description: str
    quantity: int = Field(gt=0)
    unit_price: float = Field(ge=0)

class Invoice(BaseModel):
    invoice_number: str
    total_amount: float = Field(ge=0)
    currency: str
    line_items: list[LineItem]

try:
    invoice = Invoice.model_validate(tool_call["input"])
except ValidationError as e:
    # feed e.errors() back to the model, don't just retry blind
    raise

Retry loops that actually improve the second attempt

When validation fails, a bare retry — same prompt, same schema, hope for a different roll — fixes the problem maybe half the time, because you haven't told the model what was wrong. A better loop feeds the validator's error message back into the next turn: "Your previous response failed validation: line_items.0.quantity: must be greater than 0. Correct and resend." This turns the retry into a targeted correction instead of a fresh guess, and it caps the number of attempts — two or three, not an unbounded loop — with a clear failure path (queue for human review, return a typed error) rather than silently retrying until the process times out.

Streaming makes this harder, not easier

If you stream the response for latency reasons, you no longer have a complete JSON document to hand to a validator until the stream ends — a naive JSON.parse on a partial chunk throws on every token. Either buffer the full tool-call arguments before parsing (the simplest and usually correct choice, since structured extraction is rarely latency-critical the way chat is), or use an incremental JSON parser that tolerates partial input if you specifically need to render fields as they arrive, such as a UI showing an invoice total updating live. Don't try to hand-roll partial JSON repair; it's a well-understood problem with existing libraries, and hand-rolled versions tend to silently produce wrong values instead of failing loudly.

None of this removes the need for validation — it just moves most of the failures from "malformed JSON" to "wrong value in a well-formed object," which is a much easier class of bug to catch and correct. Force the tool call, keep the schema flat and mostly-required, validate every response for real, and feed validation errors back into the retry instead of guessing again blind.