PPROMPT WORKBOOK Build a prompt
Verified 27 July 2026

The hidden cost of structured output

Structured output is sold as a pure reliability upgrade. Four studies from 2026 say it carries a real, measurable quality cost that depends on how capable the model is.

What the vendors tell you

Every major platform documents structured output as a mechanics feature: define a schema, get valid output, name your keys clearly. That is accurate as far as it goes. What none of them mention is that constraining the output shape can change the quality of the thinking that produces it.

This is one of the few places where the independent research is clearly ahead of the vendor documentation, so it is worth knowing about before you make JSON the default for everything.

Three separate findings

  1. Most of the cost comes from the instruction, not the constraint

    A 2026 study separating the two found that asking for a format in the prompt causes most of the accuracy loss, before any decoder-level constraint applies. Frontier models have largely closed this gap; it is now mainly a problem for smaller open-weight models.

  2. The remaining cost depends on model capacity

    Strong models absorb the penalty almost entirely. Weak ones do not. Even a frontier-class model was measured with a drop of around five percentage points on hard tasks under JSON constraints, and small models fare far worse: one study of sub-3B models found schema validity rose to 100 percent while answer accuracy fell from about 20 percent to 11.

  3. Valid is not the same as correct

    The same work found large numbers of outputs that were structurally perfect and semantically wrong. If you monitor only schema validity, your dashboard will look healthy while the answers get worse.

Newest and least settled. A preprint published on 20 July 2026 reports that requesting JSON also collapses answer diversity on open-ended tasks, with the most common answer's share rising sharply, and that the effect appeared with JSON and XML but not YAML or CSV. It is a single recent paper that has not been replicated, so treat it as a lead worth watching rather than an established result.

The practical rule: reason free, constrain late

The mitigation that comes up repeatedly in this research is to separate the thinking from the formatting. Let the model work the problem in open text, then produce the structured object as a second step, either in a separate call or after a thinking block within the same generation. This recovers most of the lost accuracy.

Step 1 — work the problem in plain prose.
Identify the relevant clauses, note anything ambiguous, and state your conclusion with the reason.

Step 2 — after that, and only after that, output the JSON object:
{ "conclusion": string, "confidence": "high" | "medium" | "low", "ambiguities": string[] }

Do not begin the JSON until the reasoning above is complete.

Structure it

  • Classification and extraction
  • Anything a program consumes directly
  • Fields with a fixed set of valid values
  • High-volume, repetitive work

Think first, then structure

  • Multi-step reasoning and analysis
  • Judgement calls with trade-offs
  • Anything where being wrong is expensive
  • Creative or open-ended tasks

Sources