Now that we know what an embedding is and that the best search uses a hybrid approach (combining embeddings with full-text search), it might seem that all we need to do is take the laws, create embeddings from them, and we are finished.

Fortunately, it is not that simple. Two opposing forces are at work:

• Embedding an entire document – This is the simplest solution and captures the text’s overall meaning, but it does not work in practice. If we recall how embeddings work, we know that they capture the meaning only of the text that fits within them. If we embed an entire act, the resulting vector will not properly represent any specific topic. Furthermore, the algorithm has a token limit, so the end of an excessively long document will simply be cut off.

• Embedding a single sentence – At the other end of the spectrum, we stay perfectly within the limit, but the vector contains too little information. Let us consider an example.

  “(3) The offender shall be punished by imprisonment for three to eight years if they commit the act referred to in subsection 1 and thereby cause substantial damage.”

  From this isolated sentence, we cannot determine that this is the third subsection of §224, which increases the penalty for capital fraud where the offender causes substantial damage. The increased sentencing range reflects the social harm of such conduct, particularly in relation to investor confidence and the stability of the financial market.

The solution is to choose the golden mean: provide enough context for the embedding to capture essential information while remaining within the model’s limits.

Five approaches to chunking

1. A fixed window based on token count

We divide the document into sentences and collect them until we reach, for example, 8,192 tokens (≈ 32,000 characters). Advantage – trivial implementation. Disadvantage – the beginning of a paragraph may be absent from the following chunk, distorting its meaning.

The image shows an example where the purple chunk lacks the beginning of the paragraph. As humans, we logically understand that paragraph divisions matter and that the beginning provides context for the rest of the text, so this can be a major problem.

2. Overlap

We begin a new chunk not immediately after the previous one ends, but a few sentences earlier. The green portion of the text appears in both embeddings, reducing the risk of changing the meaning by cutting it off.

However, this still does not solve the entire problem. Consider the following situation.

Although we have an overlap of 5 points, the purple chunk still lacks the beginning of the list, which is very important in this case.

We must recognize that paragraph and sentence lengths are essentially arbitrary. Consequently, overlap and splitting based on sentences, paragraphs, lists, and so forth can never entirely eliminate situations where an essential part of the text is cut off at the beginning or end.

3. Semantic chunking

Our main problem is that we want sentences that belong together to appear in a single chunk. As humans, we can naturally tell that they belong together, based on the document’s structure and meaning. Since we can capture meaning through embeddings, we can compare sentences or chunks through embeddings and extend or shorten a chunk based on their similarity.

In this way, we can improve chunking automatically because we can detect the end of a chunk by meaning, which should prevent situations like the previous one.

I say “should” because no automated approach is 100% reliable. One example of failure would be the previous list of points in §36. If the list were too long for one chunk, we would have to split it into two—and we would be back to the previous problem.

4. AI context

The problem we are trying to solve with increasingly complex approaches is that a chunk lacks context about its location in the document. For example, if it contains a list, the chunk needs information about the purpose of that list, not merely its individual items.

The theory behind AI context is to divide the document hierarchically—in our case into titles, divisions, sections, and so on. We generate a summary for each part and insert it at the beginning of the chunk to explain where that chunk is located. For our previous example involving §36, we might add the following context: “This part of the Criminal Code belongs to the general part and addresses the fundamental principles of sentencing. Specifically, it focuses on mitigating and aggravating circumstances that affect the severity and type of punishment.”

Thus, even if the list in §36 is too long, we retain information about what it concerns, and the embedding itself will be more accurate.

There are a great many approaches to chunking. We must not forget that the more complex the chosen approach, the more expensive and slower it becomes, while also increasing the likelihood of complete failure in an unexpected case. We therefore need to consider carefully which types of documents we want to chunk and apply domain expertise to the specific use case.

For example, if we know that a document contains only paragraphs that do not follow on from one another, it is sufficient to split the document into those paragraphs and add basic information about the document to each as context.

I hope this article has given you insight into the various chunking methods. The subject remains an active area of research and continues to evolve.