Lotook
Structured outputs are an LLM API feature that forces a model’s response to conform to a JSON Schema you supply. Using constrained decoding, the API compiles your schema into a grammar or state machine that restricts token generation, preventing syntax errors and missing fields across OpenAI, Anthropic, and Google APIs.
What Are Structured Outputs in an LLM API?
Structured outputs in an LLM API constrain a model’s generated tokens so the completed reply matches a developer-provided JSON Schema. Both Anthropic and Google describe the underlying mechanism as constrained decoding: before sampling begins, the API compiles your schema into a grammar or finite-state machine that masks out any next token that would violate the schema’s structure, key names, or value types.
Without structured outputs, asking a language model in a prompt to reply in JSON works inconsistently in production. Anthropic’s documentation highlights four common failure modes when relying on prompt instructions alone: parsing errors from invalid JSON syntax, missing required fields, inconsistent data types, and schema violations requiring error handling and retries. In practice, those failures look like these generic examples:
- Parsing errors from invalid JSON syntax: For example, unclosed braces, trailing commas, or markdown prose wrapped around the payload can break a JSON parser.
- Missing required fields: For example, a model might omit a key that downstream application code expects.
- Inconsistent data types: For example, a numeric field might arrive as a string, or a single value might arrive instead of an array.
- Schema violations requiring error handling and retries: For example, unrequested keys or out-of-range values force retry loops and defensive validation code.
Constrained decoding removes the need for strongly worded formatting instructions in your prompt. To keep application types aligned with the schema you pass to the API, OpenAI and Google provide native SDK helpers that accept Pydantic BaseModel classes in Python and Zod objects in JavaScript, while Anthropic’s Python SDK accepts a Pydantic model via client.messages.parse() and parses the response into that type.
How Do Structured Outputs Differ From JSON Mode and Prompt-Only JSON?
Structured outputs guarantee adherence to your exact JSON Schema, whereas JSON mode and prompt-only instructions cannot enforce specific property names, nested shapes, or required fields.
JSON mode was the intermediate step offered before full schema constraints existed. On OpenAI, enabling JSON mode via { "type": "json_object" } ensures the model produces syntactically valid JSON, assuming the request does not hit an edge case such as running out of output tokens mid-object. However, JSON mode carries explicit caveats in OpenAI’s documentation: you must include an instruction to output JSON somewhere in the conversation context, or the API throws an error when the word JSON is absent and the model can otherwise generate an unending stream of whitespace until it reaches the token limit.
OpenAI characterizes structured outputs as the evolution of JSON mode and recommends using structured outputs whenever supported, reserving JSON mode for older models such as gpt-4-turbo and earlier.
| Feature | Prompt-Only JSON | JSON Mode | Structured Outputs |
|---|---|---|---|
| Valid JSON syntax guaranteed | No | Yes, unless truncated | Yes, unless refused or truncated |
| Enforces exact schema and types | No | No | Yes |
| Requires JSON instruction in prompt | Yes | Yes, required by OpenAI | No |
| Eliminates formatting retries | No | Partially | Yes, for schema shape |
| Native Pydantic and Zod parsing | Manual | Manual | Supported by vendor SDKs |
Structured Outputs vs. Function Calling: When Should You Use Each?
Structured outputs format the model’s final reply to your application or end user, while function calling connects the model to external tools, APIs, and data sources during execution.
Developers often confuse the two features because both rely on JSON Schema definitions. Both OpenAI and Google draw the same architectural boundary:
- Use structured outputs when you want the final assistant message formatted cleanly for your UI, database, or downstream pipeline, such as data extraction or structured classification.
- Use function calling (tool use) when the model needs to request an action, look up live records, or invoke a backend routine before it can finish answering. For a deeper look at how tool definitions fit into modern protocols, see our tool calling and MCP guide.
OpenAI offers strict schemas in two distinct forms: function calling with strict schemas when connecting the model to tools or data, and a json_schema response format when structuring the reply to the user. Anthropic separates its two features into output_config.format for Claude’s final response and strict: true on individual tools to guarantee valid tool names and input arguments, and explicitly supports using both in a single request. Combining both is especially practical in AI agent workflows, where intermediate tool calls and the final summary must follow strict contracts.
How OpenAI, Anthropic, and Google Implement Structured Outputs
OpenAI, Anthropic, and Google each implement structured outputs through schema compilation as of November 2026, though their request parameters, supported JSON Schema keywords, and complexity ceilings differ.
| Provider | How to Enable | SDK Schema Helpers | Notable Schema Rules | Notable Limits |
|---|---|---|---|---|
| OpenAI | Responses API text.format or Chat Completions response_format with type: "json_schema" and strict: true |
Pydantic BaseModel (Python), Zod (JS) |
Root must be an object, not anyOf; all fields in required; additionalProperties: false required; keys emitted in schema order |
Up to 5,000 properties, 10 nesting levels, 120,000 chars across names/enums, 1,000 enum values |
| Anthropic Claude | output_config.format with type: "json_schema", plus strict: true on tools |
client.messages.parse() with Pydantic (Python) |
Supports $ref/$defs, anyOf, and allOf (with limits); additionalProperties: false required; no numerical or string length bounds |
Max 20 strict tools, 24 optional parameters, 16 union parameters, 180s compile timeout; 24-hour grammar cache |
| Google Gemini API | Response config with text type, mime_type: "application/json", and schema |
Pydantic (Python), Zod (JS) | Supports minimum/maximum, minItems/maxItems, prefixItems; strips unsupported bounds silently |
Large or deeply nested schemas fail with 400 INVALID_ARGUMENT |
OpenAI
OpenAI supports structured outputs starting with GPT-4o and recommends gpt-6-astra for new projects. Calling the API with strict: true and an unsupported keyword such as allOf, not, if/then/else, dependentRequired, or dependentSchemas returns an error rather than failing silently. OpenAI supports string formats including date-time, time, date, duration, email, hostname, ipv4, ipv6, and uuid, along with numeric bounds (minimum, maximum, exclusiveMinimum, exclusiveMaximum, multipleOf) and array bounds (minItems, maxItems).
Anthropic Claude
Anthropic supports structured outputs across all models on the Messages API, though not inside Managed Agents. When you use client.messages.parse() in the Python SDK, unsupported constraints such as minimum, maximum, minLength, and maxLength are automatically moved into field descriptions so Claude sees the intent, and the SDK validates the returned payload against your original Pydantic model locally.
Claude caches compiled grammars for 24 hours from last use; changing schema structure or the tool set invalidates that grammar cache, while editing field names or descriptions does not. Because Claude also receives an automatic system prompt explaining the JSON format, input token counts rise slightly, and changing output_config.format invalidates the conversation’s prompt cache. Required properties are emitted first, followed by optional ones. Structured outputs work with streaming, token counting, and Batch processing (with Anthropic’s 50 percent discount). However, combining JSON outputs with Citations returns a 400 error, and structured outputs are incompatible with message prefilling.
Google Gemini API
Google highlights data extraction, structured classification, and agentic workflows as primary use cases for Gemini structured outputs. Unlike OpenAI and Anthropic, Gemini silently strips and ignores several keywords without raising an error, including minLength, maxLength, minProperties, maxProperties, uniqueItems, contains, minContains, and maxContains. You should always run Pydantic model_validate_json() or Zod .parse() on Gemini responses to enforce those constraints in application code. If Gemini returns 400 INVALID_ARGUMENT for an overly complex schema, Google advises flattening nesting, reducing optional fields or enums, shortening property names, and moving non-structural rules into field descriptions.
How to Design a Reliable JSON Schema for LLMs
A reliable JSON Schema for LLM structured outputs stays flat, marks fields required, and uses nullable types instead of optional keys. Designing schemas around shared cross-vendor rules helps prevent compilation errors and unexpected runtime rejections across providers.
Use this checklist before shipping a schema across OpenAI, Claude, or Gemini:
- Keep nesting shallow: Flatten deeply nested objects where possible to stay well below OpenAI’s 10-level cap and avoid Gemini or Claude compilation rejections.
- Require every field: List every object key in
required, as OpenAI enforces this rule strictly and Claude caps total optional parameters per request at 24. - Use nullable unions for missing data: Represent optional values with
"type": ["string", "null"]rather than omitting keys fromrequired. - Lock down objects: Set
"additionalProperties": falseon every object definition, which both OpenAI and Anthropic mandate. - Constrain fixed choices with enums: Replace open-ended string categories with explicit
enumarrays when only a known set of labels is valid. - Write clear descriptions: Place domain rules and ignored constraints, such as character length targets or fallback instructions, inside each property’s
description.
Below is an illustrative example schema for classifying a customer support ticket. It follows the cross-vendor rules above: an object root, all fields required, a nullable field for optional metadata, additionalProperties: false, and an enum for routing.
{
"type": "object",
"additionalProperties": false,
"required": ["category", "urgency", "summary", "order_id"],
"properties": {
"category": {
"type": "string",
"enum": ["billing", "bug_report", "account_access", "other"],
"description": "Primary topic of the support message."
},
"urgency": {
"type": "string",
"enum": ["low", "normal", "high"],
"description": "Escalation priority based on user impact."
},
"summary": {
"type": "string",
"description": "One-sentence summary of the issue in 25 words or fewer."
},
"order_id": {
"type": ["string", "null"],
"description": "Order identifier if present in the message, otherwise null."
}
}
}What Can Still Go Wrong With Structured Outputs?
Structured outputs guarantee valid syntax against your schema, yet production applications still encounter refusals, token truncation, hallucinated values, enum casing drift, first-request latency, and untrusted user input. Treating schema compliance as syntax verification rather than truth or safety verification prevents silent failures in production.
Safety Refusals and Max-Token Truncation
Models can still refuse unsafe user-generated requests or stop mid-generation when hitting your output token ceiling. On OpenAI, a safety refusal populates a separate refusal field that does not adhere to your JSON Schema. On Anthropic Claude, a refusal returns HTTP 200 with stop_reason set to "refusal" (which is still billed) and overrides your schema; similarly, hitting the output cap returns stop_reason "max_tokens" with incomplete JSON. Always branch on refusal indicators and stop reasons before calling JSON.parse() or a validator.
Schema-Valid but Factually Wrong Values
Constrained decoding forces the model to fill your schema even when the source text lacks an answer or contradicts the schema’s assumptions. A response can pass every type and enum check while containing fabricated dates, wrong categories, or plausible-sounding guesses. OpenAI advises instructing the model in the prompt on what to return when input is incompatible with the schema, such as returning null or empty fields, and Google warns developers to validate values against business rules in code. For more on why confident outputs remain wrong, read our guide on AI hallucination rates.
Enum Casing on Claude and Compile Latency
Two operational quirks catch teams off guard during rollouts:
- Enum casing on Claude: Anthropic notes that enum casing is not guaranteed during constrained decoding, so compare returned enum strings case-insensitively in your application logic.
- First-request compilation delay: Claude adds extra latency the first time a new schema compiles into a grammar before caching it for 24 hours, and OpenAI notes a similar first-request schema-processing delay on fine-tuned models.
Prompt Injection Inside Field Values
Constraining output structure does not neutralize malicious instructions hidden inside untrusted text that gets summarized or extracted into a JSON string field. A model can dutifully return valid JSON whose summary or notes field carries injected instructions or deceptive content aimed at downstream tools or human reviewers. Treat every parsed field value as untrusted input, and review our primer on what prompt injection is before passing extracted fields into automated actions.
Frequently Asked Questions
Do structured outputs slow down LLM responses?
Structured outputs can add latency on the first request with a new schema, while subsequent cached requests run without compilation overhead. Anthropic caches compiled Claude grammars for 24 hours from last use unless you alter the schema structure or tool definitions. OpenAI notes extra first-request schema-processing latency only when using fine-tuned models.
Can an LLM still hallucinate when using structured outputs?
Yes, an LLM can still hallucinate values while returning 100 percent schema-valid JSON. Constrained decoding restricts token syntax and field types, not factual accuracy. If the input lacks required information, the model may invent values to satisfy required keys unless you allow nullable fields and instruct it when to return null.
Should I use Pydantic or Zod with structured output APIs?
Using native SDK schema helpers helps keep application types and API schemas aligned. OpenAI and Google support both Pydantic in Python and Zod in JavaScript, while Anthropic’s Python SDK supports Pydantic via client.messages.parse(). These helpers convert your models into JSON Schema, parse replies into typed objects, and validate constraints like string lengths that Gemini strips or Anthropic moves into field descriptions.
What happens if a model refuses a request with structured outputs enabled?
When a model refuses a request for safety reasons, the refusal overrides your JSON Schema. OpenAI places the explanation in a dedicated refusal field instead of following the schema, while Anthropic returns HTTP 200 with stop_reason set to "refusal". Your application must inspect those fields before attempting to parse JSON.
Why does Claude or Gemini reject my JSON Schema with a 400 error?
Both Claude and Gemini reject schemas that exceed internal grammar or state-machine complexity limits with a 400 error. Claude also returns a 400 error if you include unsupported schema features like recursive schemas or numeric bounds, or if you combine JSON outputs with Citations. Flattening nested objects, shortening names, and trimming optional or union parameters usually resolves compilation rejections.
