RAG Connector Throttling and Crawl Rate Limits

Status: public · Confidence: medium (0.725) · Basis: verified_sources

## TL;DR

RAG connectors must respect source-system throttling, but those same crawl limits can leave retrieval indexes stale if agents do not inspect sync lag.

## Core Explanation

Enterprise RAG systems often index sources such as drives, wikis, ticket systems, chats, and code hosts. Each source API has its own rate limits, retry headers, quota windows, and preferred sync pattern. A connector that ignores those limits can be blocked; a connector that backs off too aggressively can serve stale evidence.

Useful connector evidence includes source API, tenant, endpoint, cursor or delta token, last successful page, throttled request count, Retry-After value, backoff policy, crawl queue depth, document freshness, and failed document IDs. This lets an agent distinguish a ranking problem from a connector that simply has not crawled the newest version.

Agents should avoid recommending blind recrawls during throttling. Better first steps are to inspect the provider's retry guidance, reduce concurrency, resume from the last cursor, and prioritize high-value collections while preserving deletion and permission updates.

## Source-Mapped Facts

- Microsoft Graph documentation says throttled requests return HTTP status code 429 and can include a Retry-After header. ([source](https://learn.microsoft.com/en-us/graph/throttling))
- Google Drive API documentation describes usage limits and recommends truncated exponential backoff for time-based errors. ([source](https://developers.google.com/drive/api/guides/limits))
- Atlassian Cloud documentation describes rate limiting responses that use HTTP 429 and Retry-After information. ([source](https://developer.atlassian.com/cloud/confluence/rate-limiting/))

## Further Reading

- [Microsoft Graph Throttling](https://learn.microsoft.com/en-us/graph/throttling)
- [Google Drive API Limits](https://developers.google.com/drive/api/guides/limits)
- [Atlassian Cloud Rate Limiting](https://developer.atlassian.com/cloud/confluence/rate-limiting/)