Embeddings 101: How search by meaning rather than words works

Embeddings are fairly simple to describe. We can assign any text a “number” that represents its meaning. Let us look at how this actually works and what the implications are for search.

I must admit that when I said “number,” that was only a simplification and not entirely accurate. Embeddings are represented by vectors. Vectors are arrows with a direction and a length.

Because we represent embeddings as vectors, we can perform operations on them. For our purposes, these are primarily addition, subtraction, and position comparison.

We can therefore determine how close two vectors are to each other. Alternatively, we can take one vector, subtract another from it, and obtain a new vector as the result.

That concludes the mathematics section; now let us move on to examples.

The vector shown represents the word “King”—its meaning in every sense in which we understand it. A king is a man who sits on a throne, a chess piece, a playing card in poker, and so on. All these meanings are represented by the arrow shown in the image.

Let us add another arrow representing the word “Queen,” again in all its meanings.

Because we can perform mathematical operations on vectors, we can see what happens when we subtract and add concepts. If we subtract “Man” from “King” and add “Woman,” we get “Queen.” This principle works across words and captures a surprising number of relationships that we do not normally even notice.

My favorite example is that if we take “meow,” subtract “cat,” and add “Santa,” we get “ho ho ho.”

We can also take the arrow for the word “dog” and identify the closest similar arrows (words). For “dog,” these might include “dogs,” “puppy,” “pet,” “bulldog,” and so on. In the same way, we can take two words and determine how far apart they are, or how similar they are. The closer they are, the more similar they are. For example, “cat” and “dog” are much closer than “cat” and “car.”

We cannot state precisely whether the words are identical, but we can determine how close or distant they are on a scale from 0 to 1. The lower the number, the closer the words are, allowing us to say that they have similar meanings or represent a similar concept.

In our specific example, we take an entire statutory section and create a vector from it. We then store that vector. We repeat the same process for every section and text in every law. When a user searches for something, we create a vector from their query and see which vectors are closest to it. For example, if we search for “balcony as a common part of a building,” the closest text we find is subsections (1 and 3) of § 10 of Act No. 182/1993.

It should be noted, however—as the king example showed—that meaning is captured in multiple senses. Practical use therefore requires additional filtering, but we still obtain highly relevant results.

A few final notes:

• The process of converting text into a vector depends on the embedding model we use. Each model will therefore produce a different result.
• We used 2D vectors in the examples to illustrate the principle simply. In practice, however, it is impossible to locate every word accurately in such a simple space. Much larger vectors, with more dimensions, are therefore used. OpenAI models, for example, use 3,072 dimensions.
• We mentioned creating an embedding from a statutory section. This is not always possible if the section is too long—in that case, it must be divided correctly before embedding, and the search method must also be adjusted to achieve the correct results.
• Care is also needed when important information about a document’s meaning appears at its beginning. We must ensure that this information is “carried over” into passages at the end of the document. Otherwise, some subsections would not be found in a search because the vector would not contain the information from the beginning of the document.
• And much more…

I hope this has given you a good overview of embeddings and how they are used.

If you have any questions, feel free to write to me—I will be happy to answer. And if you would like to try embeddings in practice, visit app.praktik.ai.