JSON mode changes what your model answers

Asking a language model to answer in JSON changes which answer it gives, not just how the answer is wrapped. A July 2026 study ran the same 31 single-word prompts across 44 language models, once in plain chat and once with a JSON format request appended. In chat, the most popular answer accounted for 41% of responses across the panel. Under JSON, it accounted for 64%. Distinct answers per prompt fell from 52 to 36, and mean answer surprisal dropped from 1.80 to 1.58 bits. Nothing about the questions changed. The only difference was a sentence asking for {"word": "<your answer>"}.

This matters for context engineering because structured output is not an edge case. It is the default way software consumes language models: every function call, every extraction schema, every agent pipeline that parses a response. The paper puts the deployment reality plainly: software does not consume language models in prose. It asks for JSON. And the surface it asks through turns out to be measurably narrower than the chat surface where models get evaluated, compared, and ranked.

What the One-Word Census measured

The study re-ran a 31-prompt battery called the One-Word Census across 44 models spanning OpenAI, Anthropic, Google, DeepSeek, Qwen, Meta, and IBM families, accessed through a single aggregator. Thirty prompts follow one pattern, “Name a [category]”: name a color, name a tree, name a board game. The thirty-first is fully open: “Pick a word.” Every prompt requests a single word, runs with no system prompt, and samples four times per model at temperature 1.0.

The baseline condition appends only “Reply with one word only.” The structured conditions append one extra clause each, asking for the same answer delivered as JSON ({"word": "<your answer>"}), XML (<word>your answer</word>), YAML, CSV, or a bare bracket wrapper ([answer]).

The design detail that makes the result interesting: none of the primary conditions used constrained decoding, token masking, or a provider’s response_format switch. The format request was plain text in the prompt. Whatever the models did differently under JSON, they did because of words in the context window, not because a grammar forced their token choices. The format clause acted as context, and the models’ behavior shifted the way behavior shifts when any other context changes.

JSON and XML compress answers; YAML and CSV don’t

The collapse is format-specific, not a general property of asking for structure. JSON cut mean answer surprisal by 0.22 bits (p = .0002) and XML by 0.19 bits (p = .002), while YAML and CSV produced no significant change and an arbitrary bracket wrapper moved diversity the other way, up 0.13 bits.

The compression is not a subtle statistical artifact spread thinly across the panel. Six of the 44 models shifted individually and significantly toward the panel’s modal answer, and within-run resampling makes the effect concrete: when a model gave the same answer across all four chat samples for a prompt, a stable default, appending the JSON clause changed that answer 53% of the time. Half of what looked like a model’s settled preference in chat was contingent on the output format.

The bracket result is the control that gives the rest of the table meaning. Brackets are structure too: a wrapper, a compliance requirement, a parseable shape. If mere formality caused convergence, brackets should compress like JSON does. Instead they loosened answers. Whatever drives the collapse is specific to certain formats, not to the presence of formatting instructions.

Tool-use training is the likely mechanism

The formats that compress are exactly the ones models are post-trained to produce as answers. JSON and XML are the delivery formats of function calling, structured-output modes, and tool-use training. YAML and CSV appear in training corpora mostly as things models read, configuration files and data tables, rather than formats they are rewarded for speaking. The paper’s interpretation is that a JSON clause does more than specify syntax. It relocates the prompt into the model’s tool-use register, where post-training has systematically rewarded convergent, schema-safe, high-probability completions.

That reading fits the bracket reversal. A bracket wrapper is structural but appears nowhere in tool-use reward loops, so it carries no gravitational pull toward the tool-calling persona. It may even signal informality, which would explain the slight loosening.

This is the same lesson we drew from the TOON vs JSON benchmark, extended one level deeper. That study showed a format’s cost is downstream of the training distribution: a compact format is expensive if the model is not fluent in it. This study shows a format’s content is downstream of the training distribution too. The choice of output format does not just change what you pay per token. It changes which tokens you get.

Software meets models on the collapsed surface

Production systems consume models almost entirely through the surface this study shows is most collapsed. Models are evaluated, benchmarked, and vibe-checked in chat, but agents call them through function-calling APIs, extraction schemas, and MCP tool interfaces, where JSON is the lingua franca. If the two surfaces behave differently, then chat-based evaluation is systematically sampling a wider, more varied distribution than the one production traffic actually draws from.

For tasks with a single correct answer, the gap is harmless or even helpful. Extraction, classification, routing, and lookup all want convergence: there is one right invoice number, and a model that reliably lands on it is doing its job. The collapse the paper measures lives in wide answer spaces, where many answers are valid and which one you get is the product.

Those wide-answer-space tasks are more common in agent pipelines than they first appear:

• Synthetic data generation. If every sample is drawn through a JSON schema, the dataset inherits the narrowed distribution, and downstream models trained on it inherit the narrowing again.
• Survey simulation and persona work. Research that uses LLMs to simulate respondent populations is directly measuring answer distributions. A 0.22-bit compression is a confound baked into the harness.
• Brainstorming and ideation features. A product that asks for ten ideas in a JSON array is asking a measurably more conservative model than the chat model its designers played with.
• Multi-agent fan-out. Spawning five agents for diverse perspectives and collecting all five answers through the same schema pays five times for samples from a narrowed distribution.

The consensus pull also rhymes with a problem support teams already know: AI replies that all sound the same. We covered the retrieval side of that in why AI customer support replies sound generic. This result adds an output-side mechanism: a pipeline that forces every reply through a schema is nudging the model toward its most average behavior at the same time the prompt is asking for personality.

How to keep answer diversity when you need structure

The practical fix is to separate what the model decides from how the pipeline receives it. Structure is a requirement of the software, not of the decision, and the two can be decoupled.

Generate in prose, package second

Let the model answer unconstrained, then convert the answer to your schema with a second lightweight call or a plain parser. The two-step costs an extra round trip, but for diversity-sensitive tasks it is the honest price of sampling the distribution you think you are sampling. The bracket finding suggests minimal wrappers are safe if you need some delimiting for parseability without invoking the tool-use register.

Constrain format only where one answer is right

Schema-constrained output is the correct tool for extraction, routing, classification, and anything else with a verifiable answer. Convergence is the goal there, and structured context on the input side plus structured output on the answer side is a reasonable default. The mistake is carrying that default over to generation tasks where the answer space is wide and the variance is the value.

Evaluate on the surface you ship

If production asks for JSON, evaluate in JSON. A model comparison, prompt tweak, or temperature setting validated in chat may not transfer to the constrained surface, because the two surfaces are measurably different distributions. This is an easy addition to the harness described in how to measure context quality for AI: run the same eval set through the exact output contract production uses, and treat chat-mode results as a different condition rather than a proxy.

What the study doesn’t show

The result is narrower than the headline invites, and the paper is careful about it. The data is a single snapshot through one API aggregator, where the requested temperature of 1.0 is not uniformly honored across providers. Each format was tested with a single clause phrasing, so the study cannot fully separate the format’s register from the specific sentence used to request it. Scoring used exact-match normalization rather than embeddings or judges, and the diversity metrics are panel-relative: the bit values depend on which 44 models make up the reference pool.

Most importantly for practitioners, the primary measurement is prompt-only. Native enforcement paths, response_format with a JSON schema and real tool-call framing, went largely unmeasured; a supplementary schema-enforcement condition covered only the 36 models that support it, with provider-dependent implementations. Grammar-level constraint could compress answer choice more, less, or differently than a prompt clause. And because the battery is single-word prompts with wide answer spaces, the study says nothing about accuracy degradation on tasks with a right answer. The claim is not that JSON makes models worse. The claim is that JSON makes models different, and different in a consistent direction: toward consensus.

The output contract is part of the context

Context engineering usually concentrates on what goes into the window: retrieval quality, structure, token budgets, placement. This study is a reminder that the last two lines of the prompt, the ones specifying the shape of the reply, act on the model the same way everything else in the window does. A format clause is context, and models respond to it as context, shifting half of their stable defaults on the strength of one sentence.

The uncomfortable version of the conclusion: if your pipeline asks for JSON, you are not sampling the model you evaluated in chat. You are sampling a narrower one, and no amount of temperature tuning on the chat surface will tell you by how much. The only way to know what your production model does is to measure it through the contract production actually uses.

Sources: Structured Output Collapses Answer Diversity Across 44 Language Models (arXiv 2607.18476) · OpenAI: Introducing Structured Outputs in the API · Anthropic: Tool use documentation