Embedding

Computers calculate with numbers, not with words. For a program to handle language, every word first has to be translated into numbers. An embedding is exactly that kind of translation: a word becomes a long list of numbers, often several hundred or thousand. The crucial trick is that the numbers are not chosen at random. Words with similar meanings get similar number lists. “Cat” and “dog” therefore end up close together, while “cat” and “interest rate” end up far apart. You can imagine it like coordinates on a map, just with very many directions instead of only latitude and longitude.

Why meaning suddenly becomes measurable

Before embeddings, programs mostly compared texts letter by letter. A search for “bicycle” would then find no page that only mentioned “bike” or “two-wheeler.” To the computer these were completely different strings of characters, even though they mean the same thing. With embeddings, the distance between two meanings can be calculated as a number. This turns a language problem into a math problem.

That is exactly what makes embeddings a fundamental building block of almost all of today’s AI systems. A chatbot first processes the entered text as number sequences before it even begins to respond. Recommendations on streaming services work the same way: films watched by similar people end up close together. The system doesn’t need to know what a crime thriller is. It’s enough that the numbers match up.

It’s important to distinguish this from the model itself. An embedding is not an AI that answers questions, but merely a representation of data. It is the intermediate layer on which other methods are built. Anyone who knows this distinction will understand many product descriptions much better.

How the numbers come about

Nobody sets the numbers by hand. They emerge during training, that is, while a program works through very large amounts of text. The basic idea is simple: words that frequently appear in the same context probably mean something similar. “Coffee” and “tea” both often appear next to “drink,” “cup,” or “hot.” The program adjusts the numbers until it can reliably predict a word from its surrounding context.

In the process, structures emerge that nobody specified in advance. A famous example: if you take the numbers for “king,” subtract “man,” and add “woman,” you end up close to “queen.” Certain directions in the number space thus correspond to concepts like gender or plurality. Such calculations don’t always work perfectly, but they illustrate well what’s happening here.

Modern systems go a step further. They don’t always generate the same number sequence for a word, but take the sentence into account. “Bank” as in a riverbank and “bank” as in a financial institution then get different values. And embeddings have long existed not just for words, but also for entire sentences, images, sounds, or products. A common misconception is that the original text can simply be recovered from a number sequence. This only works approximately, which is why embeddings can nevertheless contain sensitive information.

Embeddings in search functions and chatbots

Embeddings are most commonly encountered in search fields that find something suitable even with imprecise phrasing. Such searches are called semantic, because they go by meaning rather than exact words. The search term is converted into numbers, and the system then looks for the closest stored number sequences. Dedicated vector databases exist for this task—storage systems designed for neighborhood searches in number spaces.

In companies, this is the foundation for chatbots that answer questions about internal documents. Manuals are broken down into sections and stored as embeddings. When a question comes in, the system retrieves the most relevant sections and presents them to the language model. In news coverage, this process is often referred to by the abbreviation RAG.

In financial and tech news, embeddings therefore mostly appear indirectly. When a provider advertises better document search or fraud detection, embeddings are often behind it. Reverse image search and product recommendations in online shops work the same way. It’s an inconspicuous technology, but one that makes a great many AI features possible in the first place.