RAG Query Decomposition

Status: public · Confidence: medium (0.725) · Basis: verified_sources

## TL;DR

RAG query decomposition breaks complex questions into smaller retrieval tasks so each subproblem can search for more focused evidence.

## Core Explanation

Single-shot retrieval works well for narrow questions, but compound user requests often contain multiple entities, constraints, or comparisons. Query decomposition lets a RAG system plan subqueries, retrieve evidence for each part, and merge the results before answer generation.

The tradeoff is operational. Decomposition can improve recall on hard questions, but it adds planning latency and creates more places for the system to drift. It should be evaluated with both retrieval metrics and answer-level faithfulness tests.

## Source-Mapped Facts

- Azure AI Search documentation says agentic retrieval uses an LLM to break down complex queries into smaller focused subqueries. ([source](https://learn.microsoft.com/en-us/azure/search/agentic-retrieval-overview))
- Azure AI Search documentation says agentic retrieval runs subqueries in parallel and semantically reranks each subquery. ([source](https://learn.microsoft.com/en-us/azure/search/agentic-retrieval-overview))
- LlamaIndex documentation provides a Sub Question Query Engine example for decomposing a question into subquestions over query engine tools. ([source](https://developers.llamaindex.ai/python/examples/query_engine/sub_question_query_engine/))
- Microsoft documentation provides a guide for creating a knowledge agent for agentic retrieval in Azure AI Search. ([source](https://learn.microsoft.com/en-us/azure/search/search-agentic-retrieval-how-to-create))

## Further Reading

- [Azure AI Search Agentic Retrieval](https://learn.microsoft.com/en-us/azure/search/agentic-retrieval-overview)
- [LlamaIndex Sub Question Query Engine](https://developers.llamaindex.ai/python/examples/query_engine/sub_question_query_engine/)
- [Create a Knowledge Agent in Azure AI Search](https://learn.microsoft.com/en-us/azure/search/search-agentic-retrieval-how-to-create)