LLM Cost and Latency Evaluation

Status: public · Confidence: medium (0.725) · Basis: verified_sources

## TL;DR

LLM evaluation should track cost and latency alongside quality because the best answer is not deployable if it is too slow or too expensive.

## Core Explanation

Production LLM systems trade off quality, latency, and cost. A larger model, longer prompt, or more retrieval context may improve answer quality while hurting response time and budget. Evaluation should therefore include token usage, time to first token, full completion latency, retry rate, and cost per successful task.

Agents can use these metrics to pick a model, decide when to compress context, or escalate slow paths. The numbers must come from measured traces or provider-specific token accounting, not assumptions.

## Source-Mapped Facts

- OpenAI latency optimization documentation describes strategies for improving response latency in API applications. ([source](https://developers.openai.com/api/docs/guides/latency-optimization))
- Anthropic documentation provides guidance for reducing latency in Claude applications. ([source](https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-latency))
- OpenAI token-counting guidance describes estimating token usage before model calls. ([source](https://developers.openai.com/cookbook/examples/how_to_count_tokens_with_tiktoken))

## Further Reading

- [OpenAI Latency Optimization](https://developers.openai.com/api/docs/guides/latency-optimization)
- [Anthropic Reduce Latency](https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-latency)
- [OpenAI Cookbook Token Counting](https://developers.openai.com/cookbook/examples/how_to_count_tokens_with_tiktoken)