Retrieval-Augmented Generation

Programs like ChatGPT generate text based on patterns they have learned from vast amounts of text. This knowledge is fixed and ends at a certain point in time. They know nothing about your school’s price list or your parents' contract. Retrieval-Augmented Generation solves this problem with an intermediate step. First, the system searches a collection of documents for relevant passages. These passages are attached to the question, and only then does the program formulate its answer. The English name literally means: text generation that is augmented by looking things up.

Why looking things up is better than memorizing

AI text programs tend to invent plausible-sounding things when they don’t know something. Experts call this hallucination. This is not strictly a bug but a consequence of how the system is built: the program always predicts the most probable next word. It does not check whether the result is correct. But if the right text passages are right there in front of it, the rate of invention drops significantly.

The second advantage is up-to-dateness. Retraining a model costs millions and takes weeks. Adding a new document to a collection takes seconds. Companies can thus use their internal knowledge without touching the AI itself. This is the main reason why almost every corporate chatbot project today is built on this principle.

Then there is verifiability. Because the system knows which document a piece of information comes from, it can provide the source along with the answer. The user can look it up and judge for themselves. With an answer drawn purely from the model’s memory, this is impossible, since there is no longer a single identifiable source.

From the question through vector space to the answer

First, all documents are broken down into small chunks, usually one to two paragraphs long. Each chunk is converted into a long sequence of numbers, called an embedding. This sequence of numbers describes the meaning of the text. Similar meanings produce similar sequences of numbers. All these sequences of numbers end up in a special database, the vector database.

When someone asks a question, it too is converted into such a sequence of numbers. The database then searches for the chunks with the most similar sequences of numbers. The advantage over a normal keyword search: what matters is the meaning, not the exact word. A question about vacation entitlement will therefore also find a paragraph that only talks about days off.

The three to five best matches are now sent to the language model together with the question. Along with this comes an instruction such as: answer only with what is stated in these texts. The model then writes a fluent answer from this. In doing so, it acts more as a formulator than as a source of knowledge. Poor matches lead to poor answers, no matter how good the model is. That is why most of the effort goes into the search, not into the text generation.

RAG in search engines, support, and corporate archives

When Google or Bing display a summarized answer with footnotes above the search results, this is exactly the principle at work. The footnotes are the retrieved sources. The feature where a chatbot summarizes an uploaded PDF file and answers questions about it also often works this way.

In professional life, the most common use is customer support. A chatbot accesses manuals and old ticket histories. Banks and insurance companies use the same method for contracts and regulations. This is why the abbreviation RAG appears very frequently in job postings and stock market news. Vector database providers have, for this reason, become companies in their own right with high valuations.

A common misconception is that RAG makes the model smarter. It doesn’t. It merely gives it better material to work with. This method must be distinguished from fine-tuning, in which the model itself is further trained on one’s own data. Fine-tuning changes style and capabilities, while RAG supplies facts. Many projects combine both.