Structured Outputs

Pattern: A named solution to a recurring problem.

Constrain a language model’s response to a known schema so the next program in the pipeline can parse it without guessing.

Also known as: Constrained Decoding (the implementation technique), Response Format, Strict Tool Use (the tool-call variant)

Understand This First

• Tool — the dominant consumer of structured outputs; tool calls only work when the model returns parseable arguments.

• Schema (Serialization) — the vocabulary (JSON Schema, Pydantic, Zod) that a structured-output contract is written in.

• Agent-Computer Interface (ACI) — the design surface where response-shape decisions are made.

Context

A language model emits text. The program that called the model usually wants something else: a tool invocation, a typed record, a list of extracted entities, a routing decision, a graded score. Somewhere between the model’s free-form text and the next program’s typed input, the gap has to close.

The original move was to ask the model nicely. “Reply with a JSON object that has these three fields.” The model would mostly comply. Then it would add a markdown code fence, apologize before answering, invent a fourth field, or omit a comma, and the downstream JSON.parse would crash. JSON mode improved matters by guaranteeing valid JSON, but it still did not guarantee that the JSON matched the requested schema. OpenAI’s 2024 launch comparison put an older model, gpt-4-0613, below 40% on its internal complex-schema eval and gpt-4o-2024-08-06 with Structured Outputs at 100%. That comparison measures two models as well as two output regimes, so it does not establish a prompt-only-versus-strict gain for the same model.

Structured outputs close that gap at the model layer. The caller declares a schema; the provider constrains ordinary generation to conform. The caller still has to handle refusals and truncated responses, but it no longer has to guess where a valid result begins or what shape it takes. The pattern is now standard across OpenAI, Anthropic, Google, Cohere, vLLM, and cross-provider routing layers such as LangChain, LiteLLM, and OpenRouter.

Problem

How do you connect a model that produces tokens to a program that needs typed values, without spending the rest of your career writing parser fallbacks?

The intermediate step has to be reliable enough that the calling code can treat the model’s response as a typed result, not as an untrusted blob to defensively parse. It has to be cheap enough to use on every call. And it has to leave the model enough room to actually think: a schema so tight that it suppresses reasoning is worse than a schema that occasionally fails.

Forces

• Reliability versus expressiveness. A strict schema rules out malformed responses, but it can also rule out useful answers the schema-author didn’t anticipate. The right shape lets the model say what it needs to say while ruling out shapes the caller can’t handle.

• Latency cost of constrained decoding. Constraining generation at the token-sampling layer adds work to each step. On short responses the cost is invisible; on long ones it shows up in the wall clock.

• Reasoning quality versus structural rigor. Tam et al. found that tighter format restrictions reduced reasoning performance. Will Kurt’s reanalysis held the prompt constant and found constrained generation matching or slightly beating unstructured generation on the same tasks. The effect is disputed, but leaving a free-form reasoning field or separating reasoning from the constrained answer is cheap insurance on tasks where the model must work toward a conclusion.

• Schema drift between client and model. When the schema lives in two places (the calling code and the request body) it will eventually fall out of sync. The team that doesn’t generate one from the other will spend an afternoon a quarter chasing the divergence.

• The wrong required field. A required field the model can’t fill cleanly produces a fabricated value rather than an honest gap. This is one of the most common ways structured outputs go wrong, and it’s invisible until you read the data.

Solution

Declare a schema, hand it to the provider, and let the provider’s constrained-decoding layer enforce the response shape. The schema is part of the request, not the prompt. The model’s natural-language instruction can still describe what to fill the fields with; the shape is no longer the model’s responsibility. Treat a refusal or a max_tokens stop as a separate outcome rather than as a schema-valid result.

Three implementation styles are common, and most production systems use a mix:

Provider-native JSON Schema. OpenAI, Anthropic, Google, and Cohere all accept a JSON Schema, or a Pydantic or Zod model that compiles to one, on the request. The provider runs constrained decoding under the hood: at each token-sampling step, candidate tokens are filtered to those that keep the response on a path that can still satisfy the schema. OpenAI’s Responses API puts the schema under text.format; Chat Completions uses response_format. Anthropic uses output_config.format for JSON output and strict: true on tool definitions. Google’s Interactions API uses top-level response_format, while generateContent retains responseSchema. Cohere carries the schema in response_format.

These providers support subsets of JSON Schema rather than the whole language. Unsupported keywords may fail the request or go unenforced. OpenAI strict mode requires every field and emulates optional fields with a union that includes null; Anthropic likewise rejects several constraint forms and requires additionalProperties: false. Design absent-data states explicitly. Otherwise a required field turns “unknown” into a fabricated value.

Tool-call schemas. Every tool the model can call is declared with an input schema. When strict tool use is enabled, the response is structurally a tool invocation: a tool name plus arguments constrained to satisfy the schema. OpenAI and Anthropic mark tools strict; Cohere uses strict_tools. Without strict mode, the schema is strong guidance plus validation, not a generation guarantee. Tool use is structured outputs in disguise: the schema lives on the tool definition rather than on the output request.

Validate-and-retry frameworks. Libraries like Instructor, LangChain’s structured output, and Pydantic AI wrap any model behind a typed interface: the caller passes a target type; the library serializes a schema, sends the request, validates the response, and retries on failure with the validation error injected back into the next prompt. This is the right answer when working across providers that don’t all support native constrained decoding, or when the schema is too dynamic to express in the provider’s format.

The cross-cutting discipline is the same in all three: schema is contract, prompt is intent. Keep the what to fill in the prompt and the how it must look in the schema. Don’t restate the shape in the prompt; the model can already see it. Don’t try to enforce shape from the prompt; the model is no longer the right enforcement layer.

Tip: Leave the model room to think. If a schema requires the model to commit to a final answer in one field with no scratch space, consider adding a reasoning, analysis, or thinking string field before the answer field. The model fills it on its way to the answer, and the cost is a few extra tokens. Whether format restrictions reduce reasoning quality depends on the task, model, and prompt, so measure the effect rather than assuming it.

How It Plays Out

A team builds an extraction pipeline that pulls structured records out of inbound contracts. The first pass uses prompt-only instruction: “Reply with a JSON object containing party_a, party_b, effective_date, term_months.” It works most of the time. Once a week, the model returns a date in a format the parser doesn’t recognize, or wraps the JSON in a markdown code fence, or apologizes that one of the fields wasn’t visible in the document. The downstream pipeline catches the parse error and retries. After three months the retry rate is 6% and the retry log is the team’s largest unread Slack channel.

The second pass switches to provider-native structured outputs. The schema declares effective_date as an ISO-8601 date string and term_months as an integer. The team adds a notes string field for the model to flag fields it couldn’t extract cleanly, replacing the missing-data fabrication problem with an honest “field not present in document” annotation. Parse-error rate drops to roughly zero. The Slack channel goes quiet.

A few weeks in, the team notices a problem they hadn’t seen before: contracts written with relative dates (“the third Tuesday of next month”) show up as fabricated absolute dates, because the schema is too tight on the date format to admit anything else. They add a date_is_relative boolean field and a relative_date_text string field; the model now surfaces the cases the parser was previously hiding.

A coding agent uses strict tool-call schemas as its primary interaction surface. Every action (read a file, run a test, search the codebase, write a patch) is exposed as a tool with a typed input schema. When the model decides to read a file, it emits a structured tool call with path, start_line, and end_line arguments that the harness can dispatch directly. The harness checks for refusal or truncation before dispatch, then records a clean tool invocation rather than interpreting free-form text. The whole stack downstream of the model gets simpler.

A generator-evaluator loop has the evaluator return a structured judgment: a numeric score (0–10), a categorical verdict (accept, revise, reject), and a free-text rationale. Without a schema, the evaluator’s responses ranged across formats; the loop spent more time normalizing the verdict than acting on it. With a strict schema, the verdict is reliably one of three enum values and the score is reliably an integer in range. The next stage of the loop can be a simple switch statement.

Example Prompt: “You are an extraction agent. The user will paste a meeting transcript. Use the extract_actions tool, which has a schema requiring action_text, owner_name, and due_date_iso. For each action item that doesn’t have a clear owner or due date, set the corresponding field to null and add a one-line note in the rationale field explaining what was unclear. Don’t fabricate names or dates.”

Consequences

The wins show up immediately. Parse-error rates collapse when a provider constrains generation against the schema. OpenAI reported 100% on its internal complex-schema eval for gpt-4o-2024-08-06 with Structured Outputs, compared with less than 40% for the older gpt-4-0613; useful evidence, but not a same-model ablation. The downstream pipeline gets simpler because malformed-JSON and wrong-shape branches disappear. It still needs branches for refusal, truncation, and semantically wrong but schema-valid data.

The cost is a discipline, not an outage. Providers may spend extra time compiling a schema on its first use; later calls can reuse the compiled grammar. Format restrictions may affect reasoning quality, so evaluate the actual task and model. The most subtle failure mode is a fabricated value for a field the model cannot honestly fill: the schema validates but the data is wrong. Make absent-data states explicit with nullable fields, a confidence field, or a structured missing_reason enum. Strict providers that require every property make this design choice unavoidable.

A second cost is the architectural commitment. Once the schema is in production, changing it has the same cost as any other API change. Versioning structured-output contracts the way you version any other interface (additive changes only, deprecate before remove, never reuse a field name with a different type) pays off as soon as more than one consumer reads the data.

A third is portability. Provider-native structured outputs work brilliantly inside one provider’s stack. Cross-provider abstractions (LiteLLM, OpenRouter) flatten the differences but at the cost of dropping to the lowest common denominator on schema features. Teams that need to swap providers at the model-routing layer eventually pick a validate-and-retry framework as the portable substrate and accept the extra round-trip cost on responses that fail validation.

Structured outputs also shrink certain attack surfaces. A response constrained to a fixed schema can’t smuggle arbitrary control-flow text into a downstream parser, which closes off some prompt-injection routes that depend on the response containing free text. They are not a substitute for output encoding at the human-facing surface, which is a separate problem with its own discipline. The schema constrains what the model can say; encoding constrains what the rendering layer can do with what was said.

Complements: MCP (Model Context Protocol) — MCP standardizes how tools are exposed; structured outputs are how the model returns parseable arguments to call them.

Complements: Verification Loop — Validation failures on structured output are the most common signal that drives the loop's next iteration.

Contrasts with: Output Encoding — Output encoding is downstream sanitization for human-facing surfaces; structured outputs constrain the model itself, upstream of any sanitizer.

Depended on by: Action-Selector — Action-Selector returns a schema-validated action ID and parameters, so structured outputs are the mechanism that makes its decode safe to execute.

Depended on by: Agent Workflow Graph — Typed edges in an agent workflow graph need outputs the next step can parse without guessing.

Depended on by: Prompt Chaining — Prompt Chaining depends on this.

Depends on: Code Mode — An agent that writes code calling tools needs structured-output discipline at every call site.

Detects: Smell (AI Smell) — When a strict schema forces the model to fabricate values for fields it cannot fill, the result is a recognizable AI smell.

Enabled by: Generator-Evaluator — An evaluator that returns a graded JSON judgment depends on the model's response conforming to a fixed shape.

Enables: Tool — Tool calls only work because the model's response can be constrained to the tool's input schema.

Mitigates: Prompt Injection — A response forced into a fixed schema cannot smuggle arbitrary control-flow text into a downstream parser.

Mitigates: Review Overcorrection — A review schema that permits an empty findings list lets a reviewer report that nothing needs fixing.

Shaped by: Agent-Computer Interface (ACI) — The shape of a structured response is one of the ACI's central design surfaces.

Uses: Schema (Serialization) — Structured outputs ride on JSON Schema, the same wire-format vocabulary used to describe data in transit.

Sources

The mechanism draws on two decades of constrained-decoding research, ported to the autoregressive language-model setting. Provider support arrived in stages: OpenAI launched Structured Outputs in August 2024; Anthropic’s structured-outputs documentation records a November 2025 public beta and February 2026 general availability; Google and Cohere expose comparable schema-constrained response formats in their current API documentation.

Brandon T. Willard and Rémi Louf’s Efficient Guided Generation for Large Language Models (2023) introduced the Outlines library and described efficient constrained generation against regular expressions and context-free grammars.

Jason Liu’s Instructor library popularized the validate-and-retry pattern in the Python ecosystem from 2023 onward. Instructor’s framing (“ask for a Pydantic model, get a Pydantic model back”) became the dominant developer-facing abstraction even in environments that later got native structured-output support, because the typed-interface ergonomics matter independently of the underlying mechanism.

JSON Schema itself is the substrate every native implementation reads. Kris Zyp’s first Internet-Draft appeared in December 2009. Later versions also appeared as Internet-Drafts, but the proposals did not become an RFC; the project decoupled its main specification work from the IETF process in 2022 while continuing to coordinate with standards work. A mature vocabulary and a decade of tooling made JSON Schema a practical choice for model outputs.

Tam et al.’s Let Me Speak Freely? (2024) reported reasoning declines under stricter format constraints. Will Kurt’s Say What You Mean reanalysis held prompts constant and found constrained generation matching or slightly beating unstructured output on its reruns. Together they make the operational rule narrower: leave room to reason when the task needs it, and test whether the extra field helps.

Further Reading

• OpenAI, “Structured model outputs” — the current request shapes, supported-schema limits, and refusal handling for OpenAI’s APIs.

• Anthropic, “Structured outputs” — JSON output, strict tool use, schema restrictions, refusal handling, and grammar-cache behavior.

• Google, “Structured outputs” — Gemini’s supported JSON Schema subset and request shapes.

• Cohere, “How do Structured Outputs Work?” — Cohere’s Chat API v2 reference for response_format and strict tool use.

• LiteLLM, “Structured Outputs (JSON Mode)” — the cross-provider abstraction’s documentation and a concise inventory of provider capabilities.

• Brian Vermeer, “Building Safer AI Agents with Structured Outputs” — how a constrained response shape narrows injection surfaces without making schema-valid content semantically safe.