What Is RAG?

Retrieval-augmented generation (RAG) is a pattern where an LLM answers using content retrieved from your own documents, rather than relying only on what it learned during training. Instead of asking the model to answer from memory, you first search a document set for relevant passages, then include those passages in the prompt.

Why use RAG instead of just prompting the model

• Your data isn't in the model. A model can't answer accurately about your internal docs, product, or recent data unless you give it that content.
• It reduces (but doesn't eliminate) hallucination. Grounding answers in retrieved text gives the model something concrete to work from.
• It scales past the context window. You can have a document set far larger than any single prompt could hold, and retrieve only the relevant slice per query.

The basic pipeline

1. Split documents into chunks (see how RAG chunking works).
2. Embed each chunk into a vector and store it in a vector database.
3. At query time, embed the user's question and retrieve the most similar chunks.
4. Send the retrieved chunks plus the question to the LLM as context.

Every one of those steps has a cost: embedding generation, vector storage, and the extra input tokens retrieved context adds to every LLM call. Use the AI Budget Planner to estimate all of them together, or see how much a RAG chatbot costs for a worked example.