Validating Structured Output
18 min
Asking a model to "return JSON with fields X, Y, Z" doesn't guarantee it actually will — a field can be missing, misspelled, or the wrong type (a string where a number was expected). Since an LLM's output is still just generated text underneath, structured output needs the same validation discipline as any other untrusted external input: check it against an expected shape before your code relies on it.
def validate_structured_output(output, schema):
errors = []
for field, expected_type in schema.items():
if field not in output:
errors.append(f"missing field: {field}")
elif not isinstance(output[field], expected_type):
errors.append(f"wrong type for {field}: expected {expected_type.__name__}, got {type(output[field]).__name__}")
return errors
This is exactly the class of bug that libraries like Pydantic and instructor exist to catch automatically at the framework level -- but the underlying check is this same idea: walk the expected schema, confirm every field exists with the right type, and surface EVERY problem found rather than stopping at the first one.
Write `validate_structured_output(output, schema)`: `schema` maps a required field name to its expected Python type. For each field in `schema`, record `'missing field: {field}'` if it's absent from `output`, or `'wrong type for {field}: expected {type}, got {type}'` if present with the wrong type. Return the list of error messages (empty if valid).
Why does asking an LLM to produce STRUCTURED output (e.g. a specific JSON shape) still require validating the result in code, rather than trusting it directly?
You can validate an LLM's structured output against an expected schema, catching missing fields and type mismatches before your code trusts the result.