LLM Evaluation Weave Evaluations and Scorers

Status: public · Confidence: medium (0.685) · Basis: verified_sources

## TL;DR

Weave evaluation evidence helps agents connect datasets, scorer definitions, model functions, traces, and metric outputs into a reproducible LLM evaluation run.

## Core Explanation

LLM evaluation can look like a single score, but the score is produced by a dataset, a model or application function, and one or more scorers. If any of those change, the result is not directly comparable to a prior run.

Agents should preserve dataset identifiers, scorer code, metric names, model function version, input and output columns, run URL, trace IDs, and UI result links. This makes it possible to tell whether a regression came from the model, prompt, scorer, dataset, or evaluation wiring.

## Source-Mapped Facts

- Weights & Biases Weave documentation describes evaluations as a way to systematically assess AI application performance. ([source](https://docs.wandb.ai/weave/guides/core-types/evaluations))
- Weave evaluation documentation says an evaluation includes a dataset, scorers, and a model or function to evaluate. ([source](https://docs.wandb.ai/weave/guides/core-types/evaluations))
- Weave evaluation documentation says evaluation results can be inspected in the Weave UI. ([source](https://docs.wandb.ai/weave/guides/core-types/evaluations))
- Weave scorer documentation describes scorers as functions or classes that evaluate model outputs and return metrics. ([source](https://docs.wandb.ai/weave/guides/evaluation/scorers))
- Weave scorer documentation says scorers can return dictionaries to log multiple metrics for one example. ([source](https://docs.wandb.ai/weave/guides/evaluation/scorers))

## Further Reading

- [Weave Evaluations](https://docs.wandb.ai/weave/guides/core-types/evaluations)
- [Weave Scorers](https://docs.wandb.ai/weave/guides/evaluation/scorers)