JSON Mode for LLM Outputs

Every agent eventually needs the model to hand back something a program can parse, not just prose. "Return JSON" in the prompt gets you close most of the time and fails silently the rest of the time — a trailing comma, an extra sentence before the brace, a field renamed to something plausible-but-wrong. Structured output stopped being a prompting trick once providers started enforcing it at the decoding level, and that shift changes how you should build the parsing layer of an agent.

Prompted JSON vs. enforced schemas

There are three tiers of reliability here, and they matter because each one fails differently. Plain prompting ("respond only in JSON") is the weakest: the model is still free-generating tokens and can wander into markdown fences, apologies, or malformed nesting under load. JSON mode (OpenAI's response_format: {"type": "json_object"}, or asking Claude for JSON with a prefilled {) constrains the output to be syntactically valid JSON but says nothing about which keys or types show up. Structured outputs with a schema — OpenAI's response_format: {"type": "json_schema", "strict": true} and Anthropic's tool-use with a matching input schema — constrain decoding token-by-token against a JSON Schema, so the model literally cannot emit a field that isn't defined or a string where the schema says integer. For anything downstream that deserializes into a typed object, use the third tier. It is not meaningfully slower and it removes an entire category of parsing bugs.

Function calling is structured output

The cleanest way to get structured data out of most agent workflows isn't a "give me JSON" instruction at all — it's a tool/function definition with the shape you want, even when you have no intention of executing anything. Defining a tool called extract_invoice with a strict parameter schema and forcing the model to call it (tool_choice: {"type": "tool", "name": "extract_invoice"} in Claude, or tool_choice: {"type": "function", "function": {"name": "..."}} in the OpenAI API) gets you the same schema-constrained decoding as JSON mode, but reuses the tool-calling machinery you already have for real agent actions, and keeps the model's natural-language reasoning separate from the payload in the response.

from anthropic import Anthropic

client = Anthropic()
schema = {
    "name": "extract_invoice",
    "description": "Extract structured fields from an invoice.",
    "input_schema": {
        "type": "object",
        "properties": {
            "vendor": {"type": "string"},
            "invoice_number": {"type": "string"},
            "total_cents": {"type": "integer"},
            "line_items": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "description": {"type": "string"},
                        "amount_cents": {"type": "integer"}
                    },
                    "required": ["description", "amount_cents"]
                }
            }
        },
        "required": ["vendor", "invoice_number", "total_cents"]
    }
}

resp = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    tools=[schema],
    tool_choice={"type": "tool", "name": "extract_invoice"},
    messages=[{"role": "user", "content": invoice_text}]
)
data = next(b.input for b in resp.content if b.type == "tool_use")

Validate anyway, even with strict mode

Strict schema enforcement guarantees the shape is right; it does not guarantee the values are sane. A strict schema will happily let the model return total_cents: 0 for an invoice it couldn't actually read, or a line_items array that doesn't sum to the total. Run the output through your normal validation layer (Pydantic, Zod, whatever your stack uses) for business-rule checks — required-if-present fields, cross-field consistency, range checks — the same way you would validate a POST body from an untrusted client. The model is exactly that: an untrusted client that happens to be very good at guessing what you want.

Streaming and partial JSON

If you stream the response for latency reasons, you receive a JSON document one token at a time, which means naive json.loads() on every chunk fails until the very last token arrives. Use a incremental/partial JSON parser (most SDKs ship one, or a library like json-repair for ad hoc cases) if you want to show a UI field filling in progressively, and only treat the parse as final once the stream's stop event fires. Don't try to hand-roll bracket counting — nested strings containing braces break it in ways that are annoying to debug.

Retry and repair, not silent failure

Even with strict schemas, calls fail: the model can hit max_tokens mid-object and truncate valid JSON, or the schema itself can be too permissive and let through a technically-valid-but-useless response (an object with every optional field null). Have a repair path — a single retry with the parse error appended to the context ("your last response failed to parse: . Return only valid JSON matching the schema.") resolves the large majority of truncation and malformation issues. Log both the raw output and the validation error on every failure; without that you're debugging a black box after the fact.

Wrapping up

Use schema-constrained decoding (structured outputs or forced tool calls) as the default, not free-text prompting for JSON — it's supported by every major provider now and removes most of the parsing failure surface for free. Then validate the result anyway, because a well-shaped object can still be a wrong one, and build one retry-with-error-feedback path before you consider the extraction pipeline production-ready.