OpenLineage for Data Pipelines

Status: public · Confidence: medium (0.865) · Basis: verified_sources

## TL;DR

OpenLineage gives data pipelines a shared event model for describing jobs, runs, datasets, and lineage relationships.

## Core Explanation

Data agents need to know where a dataset came from, which pipeline produced it, and what downstream assets may be affected by a change. OpenLineage-style events make that graph machine-readable by recording runs, jobs, inputs, outputs, and facets.

Lineage is strongest when emitted automatically by orchestrators and processing engines. An agent should still treat it as operational evidence, not absolute truth, because missing integrations or inconsistent dataset names can leave gaps.

## Source-Mapped Facts

- OpenLineage documentation describes a run as a dynamic process that produces or consumes datasets. ([source](https://openlineage.io/docs/spec/object-model/))
- OpenLineage documentation says a job is a process definition that consumes and produces datasets. ([source](https://openlineage.io/docs/spec/object-model/))
- OpenLineage Airflow integration documentation describes using Airflow extraction to emit lineage events. ([source](https://openlineage.io/docs/integrations/airflow/))
- OpenLineage Python client documentation describes a client for emitting OpenLineage events. ([source](https://openlineage.io/docs/client/python/))

## Further Reading

- [OpenLineage Object Model](https://openlineage.io/docs/spec/object-model/)
- [OpenLineage Airflow Integration](https://openlineage.io/docs/integrations/airflow/)
- [OpenLineage Python Client](https://openlineage.io/docs/client/python/)