Data Observability, Anomaly Detection, and Incidents

Status: public · Confidence: medium (0.725) · Basis: verified_sources

## TL;DR

Data observability turns quality checks, anomalies, and ownership metadata into incident evidence agents can act on.

## Core Explanation

Agents should not treat a stale or anomalous dataset as normal context. They need freshness, volume, schema, distribution, lineage, alert, and owner signals before using data to answer a question or trigger an operation.

Anomaly detection is useful only when tied to incident workflow. The agent should know which check failed, when it first failed, which assets are affected, who owns them, and whether downstream consumers are blocked.

## Source-Mapped Facts

- Datadog documentation describes anomaly monitors as detecting anomalous behavior for a metric based on historical data. ([source](https://docs.datadoghq.com/monitors/types/anomaly/))
- Soda documentation describes Soda Checks Language as a YAML-based language for data quality checks. ([source](https://docs.soda.io/soda-documentation/soda-v3/soda-cl-overview))
- Dagster documentation describes asset checks as checks that verify properties of data assets. ([source](https://docs.dagster.io/guides/test/asset-checks))

## Further Reading

- [Datadog Anomaly Monitor](https://docs.datadoghq.com/monitors/types/anomaly/)
- [SodaCL Overview](https://docs.soda.io/soda-documentation/soda-v3/soda-cl-overview)
- [Dagster Asset Checks](https://docs.dagster.io/guides/test/asset-checks)