LLM Evaluation Tool-Call Accuracy and Argument Validation

Status: public · Confidence: medium (0.825) · Basis: verified_sources

## TL;DR

Tool-call evaluation should score both routing accuracy and argument validity: whether the model selected the right tool and whether its arguments satisfy the declared schema and task intent.

## Core Explanation

A tool-calling model can fail in several distinct ways. It can choose the wrong tool, omit a required argument, hallucinate an unsupported argument, use a semantically wrong value, or produce a schema-valid call that is unsafe for the current task.

Agents and evaluators should log expected tool, actual tool, argument JSON, validation errors, execution status, and side-effect class. Schema validation is necessary, but it does not replace semantic checks against the user request.

## Source-Mapped Facts

- The MCP Tools specification says a tool definition includes a unique name, description, inputSchema, optional outputSchema, and optional annotations. ([source](https://modelcontextprotocol.io/specification/2025-06-18/server/tools))
- The MCP Tools specification says clients invoke a tool by sending a tools/call request with a tool name and arguments. ([source](https://modelcontextprotocol.io/specification/2025-06-18/server/tools))
- JSON Schema Validation 2020-12 says validation keywords in a schema impose requirements for successful validation of an instance. ([source](https://json-schema.org/draft/2020-12/json-schema-validation))

## Further Reading

- [MCP Tools Specification](https://modelcontextprotocol.io/specification/2025-06-18/server/tools)
- [JSON Schema Validation 2020-12](https://json-schema.org/draft/2020-12/json-schema-validation)