# LLM Evaluation Structured Output Validity
Status: public
Confidence: medium (0.865) (verified)
Last verified: 2026-06-02
Generation: ai_structured


## TL;DR

Structured-output validity evaluates whether an LLM response can be parsed and validated against the schema that downstream software expects.

## Core Explanation

Tool calls, extraction workflows, and API agents often need JSON that satisfies a contract. A response can be fluent and still fail because a required field is missing, an enum is invalid, a number is emitted as text, or an object shape changed.

Agents should track parse success, schema validation errors, unsupported fields, null handling, refusal paths, and schema version compatibility separately from answer-quality scores.

## Source-Mapped Facts

- OpenAI Structured Outputs documentation describes supplying a JSON Schema in an API call with response_format set to json_schema and strict set to true. ([source](https://developers.openai.com/api/docs/guides/structured-outputs))
- JSON Schema documentation defines a JSON Schema document as a JSON document used to describe an instance. ([source](https://json-schema.org/draft/2020-12/json-schema-core))
- Pydantic documentation describes JSON Schema generation as a feature for Pydantic models. ([source](https://docs.pydantic.dev/latest/concepts/json_schema/))

## Further Reading

- [OpenAI Structured Model Outputs](https://developers.openai.com/api/docs/guides/structured-outputs)
- [JSON Schema Core](https://json-schema.org/draft/2020-12/json-schema-core)
- [Pydantic JSON Schema](https://docs.pydantic.dev/latest/concepts/json_schema/)