The problem with unstructured LLM output
LLMs produce text. Parsing that text into a data structure requires either trust (the model produces valid JSON) or defensive code (parse failures handled gracefully). Neither is a great foundation for production systems. Structured outputStructured OutputConstraining a language model to produce output that conforms to a predefined format or schema, such as valid JSON, rather than free-form text.Learn more → features enforce the format at the generation level, making parsing failures essentially impossible.
Three approaches exist: (1) promptPromptThe input text sent to a language model — the question, instruction, or context that triggers a response.Learn more →-only — ask the model for JSON and hope; (2) JSON modeJSON ModeA model setting that constrains LLM output to syntactically valid JSON format, ensuring parseable responses without full schema validation.Learn more → — guarantee valid JSON but not a specific schema; (3) JSON Schema enforcement — guarantee output matches your exact schema. Use approach 3 whenever possible.
OpenAI and Anthropic structured outputs
OpenAI: pass `response_format={"type": "json_schema", "json_schema": {"name": "result", "strict": true, "schema": {...}}}`. With `strict: true`, the model is constrained to produce output that exactly matches your schema — no additional fields, no missing required fields. This is guaranteed at the grammar level, not just prompted.
Anthropic: use the `tools` parameter with a single tool whose `input_schema` matches your desired output structure. Force the model to call the tool with `tool_choice={"type": "tool", "name": "your_tool"}`. The tool input is always valid JSON matching your schema.
Use Instructor for any provider
Instructor is a Python library that patches OpenAI and Anthropic clients to return Pydantic models instead of raw JSON: `pip install instructor`. Then: `import instructor; from pydantic import BaseModel; client = instructor.from_openai(OpenAI()); class User(BaseModel): name: str; age: int; user = client.chat.completions.create(model='gptGPTGenerative Pre-trained Transformer — the model architecture and family name behind OpenAI's most famous models, from GPT-2 to GPT-5.Learn more →-4o-mini', messages=[...], response_model=User)`. `user.name` and `user.age` are populated and type-safe.
Instructor handles retries automatically. If the model produces malformed output, it feeds the validation error back to the model and asks it to fix the response — up to a configurable maximum number of retries. This achieves near-100% reliability even with models that don't natively support structured output.
Grammar-based sampling for local models
For local models served by llama.cpp or Ollama, use grammar-based constrained generation. This works at the samplingSamplingThe method used to select each next token from a probability distribution during text generation, controlling the randomness and creativity of model outputs.Learn more → level — invalid tokens are assigned zero probability, so the model literally cannot produce invalid output.
Ollama supports JSON format natively: add `format: 'json'` to your API request. For a specific schema, use the `format` field with a JSON Schema object: `{"format": {"type": "object", "properties": {"name": {"type": "string"}}, "required": ["name"]}}`. This guarantees schema-valid output regardless of model size.