Getting reliable JSON out of a language model

“Return only JSON” is an instruction, not a guarantee. The model complies and then adds “Sure, here is the result” — and grammatically speaking, that sentence alone means the reply is no longer valid JSON. Once you accept that, the extraction code has only one sensible shape: stop trying to make the reply clean, and find the clean part inside it.

Why replies arrive wrapped

Models are trained to be helpful assistants, and assistants explain themselves. So even when you explicitly ask for JSON only, they tend to add a lead-in or a summary. That is not instruction failure; it is two objectives pulling against each other.

The other two wrappers come from formatting: Markdown code fences, and line breaks added for readability. The first is a display habit, the second is pure typesetting.

Two ways the parsing goes wrong

Most extraction code falls into one of two traps.

The first is parse directly: trim the reply and hand it to a JSON parser. The moment the model adds a lead-in, this breaks — and the error you get rarely points at the real cause.

The second is delete characters from the left: wrap the parse in a try/catch and drop one character at a time until it succeeds. This works most of the time, until one day it returns a truncated prefix that happens to be valid. Bad data now passes silently, which is far harder to debug than an outright failure.

Scan the structure, ignore the format

The more robust approach ignores formatting entirely and looks at bracket structure:

1. Scan left to right for the first { or [. That is the candidate start.
2. Begin counting nesting depth, decrementing on the matching close.
3. Skip characters inside string literals — otherwise a brace in your content skews the depth counter.
4. Handle escape sequences so an escaped quote does not terminate a string early.
5. When depth returns to zero, that span is the first balanced JSON value.

The algorithm does not care about fences, about what was said before, or about indentation. This is the scan the JSON extractor runs.

Extract, but do not repair

Draw one line clearly: the extractor extracts; it does not repair. A missing comma is still missing. A missing quote is still invalid.

The value of that line is that failure becomes explicit. The dangerous output of a tool that “helpfully fixes things” is not an error — it is a field you never asked for, silently invented. If it cannot be extracted, say so, and let the caller decide.

Reliable structured output comes from tolerant extraction, not from stronger prompting. Putting “find the first balanced value” into the parsing layer holds up far better than repeating the format instruction in the prompt.