# Pairwise LLM Evaluation
Status: public
Confidence: medium (0.725) (verified)
Last verified: 2026-06-02
Generation: ai_structured


## TL;DR

Pairwise LLM evaluation compares two outputs for the same task and asks which one is better under a rubric.

## Core Explanation

Some LLM quality differences are easier to judge comparatively than absolutely. A pairwise eval can compare a baseline prompt against a candidate prompt, one model against another, or one retrieval configuration against another. The judge can be human, model-based, or a mix of both.

Pairwise evaluation works best when the rubric is explicit and the dataset reflects product-critical tasks. It should not be treated as a universal score; ties, ordering effects, and judge-model drift need to be tracked.

## Source-Mapped Facts

- LangSmith documentation describes pairwise evaluation as comparing two application outputs for the same input. ([source](https://docs.langchain.com/langsmith/evaluation-approaches))
- OpenAI evals documentation describes evals as tasks used to measure model behavior and compare performance across models and prompts. ([source](https://developers.openai.com/api/docs/guides/evals))
- Phoenix LLM evals documentation describes evaluations that use LLMs to score or classify application traces and outputs. ([source](https://arize.com/docs/phoenix/evaluation/llm-evals))

## Further Reading

- [LangSmith Evaluation Approaches](https://docs.langchain.com/langsmith/evaluation-approaches)
- [OpenAI Evals](https://developers.openai.com/api/docs/guides/evals)
- [Phoenix LLM Evals](https://arize.com/docs/phoenix/evaluation/llm-evals)