Back to blog

Structured output across LLM providers: what actually differs

json_object, strict json_schema, and tool-schema constraints are three different guarantees — and they break on different providers. The round-trip test matrix and the patterns that survive switching.

Mappace Team · Engineering2026-08-065 min read
GuidesCompatibility

Three request shapes can express the same intent — "give me this JSON back" — and they carry three different weights of guarantee:

  • response_format set to "json_object" — "return JSON." No shape is promised. The schema you wanted is a hint you are betting on; the model decides whether to honor it.
  • response_format set to "json_schema" with strict: true — the provider constrains its decoding to your schema. The strongest guarantee on the wire — and only some models carry it at all, and "strict" is not uniform between providers.
  • The tool-call path — tools with a schema plus a forced tool_choice. The "shadow" strict mode: on many models the tool-schema path is where hard constraints actually live, while the plain json_schema path on the same model is weaker — or absent.

The first two look identical provider to provider. The per-provider support matrix for them, and the relationship to the third shape, is where production breaks.

What actually differs between providers

Gateway issue trackers are a map of the territory, because every entry is a shape that survived the provider and died in the hop:

  • Fields silently dropped at a proxy hop. A strict flag or the entire response_format object that middleware did not forward. One LiteLLM issue and its sibling tracking adapter gaps document the pattern: the direct call passes, the same request through the hop comes back unstructured.
  • Strict support that is tool-scoped. Some models accept constrained decoding only through the tool schema; asking for the same constraint via response_format is either unsupported or supported more weakly — the edge of that matrix is one issue and another.
  • "False confidence." A gateway that translates between provider dialects can swap a strict request for a non-strict one in a normalization that looks lossless. The developer sees valid JSON; the shape is the model's free choice. There is a HN discussion of exactly this: tests pass, production wobbles, and the hidden variable is a hop your test matrix never asserted on.

The common thread: the guarantee you bought (strict decoding) and the guarantee you received (maybe JSON) diverged across a layer you did not test directly.

The round-trip test matrix

The fix is unglamorous: assert the request, not only the response. For each model × shape combination you depend on:

  1. Send the strict request directly to the provider. Verify the response schema. This is the baseline.
  2. Send the identical request through every hop in the production path — SDK wrapper, gateway, retry layer. Capture what the upstream actually received (a debug echo, raw request logging, or a canary field inside the schema) and verify that the response_format or tool parameters arriving are the ones you sent.
  3. Anything that survives step 1 and dies in step 2 is a hop regression, not a model regression. Treat it as a bug in the path: fix the hop, or accept the lower guarantee in that path — deliberately, not by surprise in production.

One test per model × shape combination, run on every deploy of any component in the path. Unpleasant and mechanical, yes — that is exactly why it converts the incident into a red build.

Defensive patterns that survive switching

If the matrix is the ceiling, these are the floor:

  • Always validate the response. Parse the JSON, then schema-check the parsed value against your schema — regardless of what the endpoint claims to guarantee. This one habit removes entire categories of "it worked in staging."
  • Ship the schema subset. Define the constraint set your weakest target path supports, and use it everywhere. Escalate per-target strictness only where the matrix proves it holds. Fewer guarantees per call, but every one of them is real.
  • A fallback ladder, designed into the contract. Strict, then tool-constrained, then json_object plus repair. The repair only counts if it is designed: on parse failure, one retry with the validation error appended to the prompt — what worked, plus what the checker rejected — a budgeted, capped-at-one retry, then a hard failure. A ladder in the contract is a mechanism. A "well, retry it" loop is traffic with extra spend.

The pattern nobody gets right is the first one, because the other two feel more sophisticated.

Where the switching cost lands

Choosing a provider — or moving between them — ends up depending less on "does it support json_schema" than on "does the full path I actually run keep honoring the shape I send." One compatible request format makes the request side a one-line change; the residual risk is on the constraint side: does the hop still carry it. The exact request semantics are in the docs — run the matrix before you trust any short answer.