RAG Ingestion Error Queues and Dead-Letter Documents

Status: public · Confidence: medium (0.725) · Basis: verified_sources

## TL;DR

Dead-letter documents help RAG systems preserve failed ingestion evidence instead of silently dropping bad source files, parser errors, or index writes.

## Core Explanation

RAG ingestion pipelines often move documents through loaders, parsers, chunkers, embedders, metadata enrichers, and vector upserts. Any step can fail while the rest of the batch succeeds. Without a dead-letter path, operators may only see that an index is stale or incomplete.

An ingestion error queue should preserve the source document ID, version, connector checkpoint, parser error, chunking state, embedding model, target namespace, retry count, and whether the document had already produced indexed chunks. That context lets an agent distinguish transient delivery failure from a malformed document, ACL mismatch, unsupported file type, or unsafe reprocessing case.

The safest recovery path is not "retry everything." Agents should inspect the failed payload and retry policy first, then decide whether to reprocess, quarantine, delete stale chunks, or escalate to a data owner.

## Source-Mapped Facts

- Amazon SQS documentation says dead-letter queues isolate messages that were not processed successfully so applications can debug why processing failed. ([source](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html))
- Google Pub/Sub documentation says undeliverable messages can be forwarded to a dead-letter topic after an approximately configured number of delivery attempts. ([source](https://cloud.google.com/pubsub/docs/dead-letter-topics))
- Amazon EventBridge documentation says a dead-letter queue can retain failed events so the underlying issue can be resolved and events can be processed later. ([source](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rule-dlq.html))

## Further Reading

- [Amazon SQS Dead-Letter Queues](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html)
- [Google Pub/Sub Dead-Letter Topics](https://cloud.google.com/pubsub/docs/dead-letter-topics)
- [Amazon EventBridge Dead-Letter Queues](https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rule-dlq.html)