How to Get Structured JSON Output from LLMs: A Troubleshooting Guide

A model that tops every reasoning benchmark you can name will still hand your parser a syntax error something like one time in twenty, just from being asked to “return JSON.” This holds for GPT-4-class and Claude-class models alike, and the raw capability of the model is rarely the bottleneck. The bottleneck is the absence of any mechanism enforcing output shape at generation time — the model is predicting the next plausible token, not running a validator against your schema before it commits to a character.

That gap between “usually valid JSON” and “always valid JSON” is exactly where production pipelines break. A JSON.parse() call downstream doesn’t care about benchmark scores. It cares about one stray trailing comma, one unescaped quote, one markdown code fence the model decided to wrap around its answer out of habit. If you’re building anything that consumes LLM output programmatically — a data extraction pipeline, an agent that calls tools, a form-filling assistant — this failure mode eventually shows up in your error logs, usually at the worst possible time.

What follows is a ranked breakdown of the fixes I’ve used, ordered from most reliable to least, with the trade-offs that come with each. Most teams start at the bottom of this list and work their way up only after production breaks.

Five Fixes, Ranked by How Reliably They Close the Gap

1. Schema-Constrained Decoding (Native Structured Output Modes)

This is the only method on this list that enforces correctness at the token-sampling level rather than hoping the model complies. When you pass a JSON schema through a provider’s structured-output feature — OpenAI’s response_format with json_schema and strict: true, or the equivalent constrained-generation modes from other vendors — the API restricts which tokens are even eligible for sampling at each step. Invalid syntax isn’t discouraged; it’s removed from the probability distribution entirely.

Reliability here approaches 100% for well-formed schemas. The trade-offs are real, though: deeply nested or recursive schemas can hit provider-side complexity limits, grammar compilation adds a small amount of latency on the first request, and not every model or provider supports this yet. If your schema is flat-ish and your provider supports it, this should be your default, not your last resort.

2. Tool or Function Calling with a Strict Parameter Schema

Mechanically adjacent to method one, but framed differently. You define a function with a JSON schema describing its arguments, force tool_choice to that specific function, and the model emits its response as structured arguments rather than free text. Some earlier implementations of this were looser than dedicated JSON modes, but current versions from major providers now apply comparable grammar constraints under the hood.

This approach fits naturally into agent frameworks where JSON output is really a side effect of the model “taking an action,” rather than the primary deliverable. If your architecture is already built around tool calls, you get schema enforcement for free instead of bolting on a separate JSON mode.

3. Few-Shot Examples Demonstrating the Exact Target Schema

No API-level enforcement here — this relies entirely on in-context pattern matching. Show two or three complete input/output pairs using the precise key names, nesting, and data types you want, and the model tends to mimic that structure closely. Reliability drops noticeably compared to methods one and two, but this is the only option that works uniformly across models without native structured-output support, including open-weight models run locally where there’s no schema API to call.

The recurring failure here is subtler than outright invalid syntax: the model matches your keys and types but still wraps the whole thing in a markdown fence, or prepends “Here’s the JSON:” even when none of your examples did that. Few-shot prompting shapes content well; it’s less reliable at suppressing conversational habits layered on top.

4. Validate-and-Retry Loop

Treat the first generation as a draft, not a deliverable. Run the output through a schema validator — jsonschema in Python, ajv or Zod in JavaScript — and if it fails, feed the specific validation error back into a follow-up call: “Your previous response failed validation with this error: [error]. Return corrected JSON only, with no other text.” This closes most of what methods one through three miss, at the cost of one extra round trip of latency and tokens per failure.

I’d treat this as a mandatory fallback layer regardless of which primary method you’re using higher up this list. Constrained decoding prevents malformed syntax, but it won’t stop a model from returning a semantically valid string where an enum value was required. Validation catches problems that live at the schema-semantics layer, not just the JSON-syntax layer.

5. Prompt-Only Instructions (“Respond only in JSON”)

The weakest tier, and the one almost everyone starts with before hitting a production incident. No grammar enforcement, no examples, no validation — just an instruction buried in the system or user prompt. This is where you see the full range of failure modes: preamble sentences before the JSON, markdown fences wrapping it, truncation near the token limit that leaves a dangling brace, single quotes bleeding in from training data that looked more like Python dicts than JSON.

It’s included here mainly as the baseline everyone regresses to under time pressure, and as the clearest illustration of why the methods above exist.

Comparing the Five at a Glance

In practice, the most durable pipelines stack these rather than picking one. Method one or two as the primary mechanism, method four as a safety net underneath it, because no single layer catches every failure mode on its own.

Common Failure Modes and Where They Originate

Beyond picking the right method, it helps to know what specific symptom points to what specific cause:

• Markdown fences around otherwise-valid JSON. The model was trained on enough JSON-in-chat-responses examples that wrapping in ```json blocks became a strong prior. Strip fences with a regex before parsing, or set an explicit structured-output mode that suppresses this behavior at the source.
• Truncated output mid-object. Usually a max_tokens ceiling hit before the model finished. Either raise the token budget with margin for your largest expected schema, or instruct the model to produce compact, non-indented JSON, which uses meaningfully fewer tokens than pretty-printed output.
• Trailing commas or single quotes. Often traceable to few-shot examples that were copy-pasted from a Python REPL’s dict representation rather than true JSON. Audit your examples for this before assuming the model invented the error.
• Enum drift. The model returns "yes" where your schema expects a boolean, or a near-miss string where an exact enum value was required. Constrained decoding prevents the syntax-level version of this but not always the semantic version — worth an explicit validation check regardless.
• Fields silently omitted instead of set to null. If your schema marks a field optional, the model may drop it entirely rather than include it with a null value, which breaks code expecting the key to always be present. State explicitly in the schema and prompt whether omission or null is the expected behavior for absent data.

Most teams don’t need all five methods running simultaneously — they need the two-layer combination of constrained decoding (or tool calling) plus a validation fallback, applied consistently instead of ad hoc. If your current pipeline is stuck at method five and failing intermittently in ways that are hard to reproduce, that inconsistency is itself the diagnostic signal: it means nothing in your stack is actually enforcing the shape you’re assuming downstream.

What’s the specific failure your parser keeps hitting — malformed syntax, a missing field, or something stranger further down the schema?