Retrieval-augmented generation (RAG) is a way of building AI applications so that, before a language model answers a question, the system searches a set of approved sources, such as your policies, product documentation or support tickets, and hands the most relevant passages to the model along with the question. The model then writes its answer using that material, and many systems show which sources it drew on. RAG is how many company chatbots, internal assistants and “chat with your documents” tools give answers about your business rather than only what the model learned in training.
At a glance
- RAG combines a search step (retrieval) with a large language model (LLM) that writes the answer (generation).
- The model itself is not retrained; current content is supplied with each question, so updates can show up once the content is re-indexed.
- Answers can cite their sources, which makes them easier to check, though errors are still possible.
- Results depend heavily on the quality, freshness and permissions of the underlying content.
- Many AI assistants and enterprise search products use RAG behind the scenes, often without using the name.
What problem it solves
A language model on its own knows only what was in its training data, which stops at a point in time and does not include your internal documents. Ask it about your return policy, your network design or last quarter’s pricing and it either says it doesn’t know or produces a plausible guess, an AI hallucination. Retraining or fine-tuning a model every time content changes is slow and costly, and it makes it hard to see where an answer came from.
RAG addresses this by keeping knowledge outside the model. The model is asked to answer from the passages it is given, so answers can reflect current content and point back to a source a person can open. For a mid-market company, that is usually the practical route to an AI assistant that knows about the business.
How it works
Preparing content. Documents are collected from approved sources, split into smaller passages and indexed for search. Many systems convert each passage into an embedding, a list of numbers representing its meaning, and store it in a vector index; others use keyword search or both.
Retrieving. When a user asks a question, the system searches the index for the passages most likely to help. Well-built systems apply the user’s access permissions at this step, so people only get answers drawn from content they are allowed to see.
Generating. The question and the retrieved passages are sent to the language model with instructions to answer from that material. The model writes the response, and the application often adds links or citations to the source passages.
Keeping it current. Connectors re-index content as it changes. How often that happens, and whether deleted or restricted content is removed promptly, varies by product and configuration.
The weak points are usually retrieval rather than the model: if the right passage isn’t found, the answer suffers. Retrieved text can also carry hidden instructions, a form of prompt injection, so content sources need the same care as any other input.
When it matters for buyers
- When rolling out an internal assistant or knowledge search. Ask whether and how it retrieves your content, and from which systems.
- When content is sensitive. Permission handling at retrieval time decides whether the tool can expose HR, finance or customer data to the wrong people.
- When choosing where it runs. RAG can run on a public AI service or in a private AI environment; the index itself holds copies of your content and needs the same protection.
- When answers must be checkable. Customer-facing or regulated uses benefit from citations and logging.
- When your content is messy. RAG exposes outdated and conflicting documents, so data governance work often comes first.
Our artificial intelligence overview covers providers that build and host RAG-based tools.
Questions to ask vendors
- Which of our systems can you connect to, and how often is content re-indexed?
- How do you enforce our existing access permissions when retrieving content?
- Where are the index, embeddings, prompts and logs stored, and for how long?
- Do you or your model provider keep or train on our content or questions?
- Can users see the sources behind each answer?
- How do you measure answer quality, and what happens when retrieval finds nothing relevant?
- How do you protect against instructions hidden in retrieved documents?
How it differs from fine-tuning
Fine-tuning trains an existing model further on your examples, changing the model’s behavior or style. It is useful for teaching a format, tone or specialized task, but it is a poor way to keep facts current, because every change needs another training run and the model cannot easily show where a fact came from. RAG leaves the LLM unchanged and supplies facts at question time, so content can be updated and permissioned independently. Many production systems use RAG for knowledge and, where needed, fine-tuning for behavior. Both sit within the broader field of generative AI.