Retrieval Augmented Generation

You already have the hard idea from the last chapter:

• You cannot paste thousands of documents into every request
• You can turn text into embeddings and keep passages that sit near the question

Now finish the path: retrieve those passages, then generate an answer from them.

That whole pattern has a name: RAG — Retrieval Augmented Generation.

Plain meaning: look up first, then write.

The same story, one step further

Person’s question: “How do we reset a contractor’s laptop?”

Your library still has thousands of pages. Somewhere: wipe a loaner computer before return…

Without retrieve, the model guesses from general training.
With retrieve, it can ground the reply in your handbook.

Why split documents (chunks)

A whole PDF is often too big to embed and rank as one blob — and too big to paste even when it “wins.”

So you split long files into shorter passages (also called chunks).

Good enough for learning: split on headings or short paragraphs. Keep each passage about one idea.

Offline (once):

documents → split into passages → embed each → store text + numbers

At question time you mostly embed the question, compare, and keep the top few passages.

Retrieve: pick the closest passages

You already know the ranking idea:

1. Embed the question
2. Compare to stored passage embeddings (cosine similarity)
3. Keep the top matches (often 2–5)

  question → embedding
                 │
                 ▼
        compare to stored
        passage embeddings
                 │
                 ▼
        top passages (text)

Example:

For tiny demos, a list in memory is enough. For thousands of passages, people store vectors in a vector database so search stays fast. Same idea either way: near numbers → keep that text.

Augment: put passages in the prompt

Now build the chat prompt so the model sees the retrieved text.

Idea:

Use only the passages below to answer.
If the answer is not there, say you do not know.

Passages:
1) Wipe a loaner computer before return: sign out, erase with WipePro, tag ready.
2) …

Question: How do we reset a contractor’s laptop?

That is augmentation: you grew the prompt with evidence.

Generate: answer from the evidence

Send that request to your chat model (the one that writes replies — not the embedding model).

The model should:

• Use the passages
• Avoid inventing company rules that were not retrieved
• Prefer “I don’t know from these docs” over a confident guess

Worked flow:

If retrieve picked the wrong passage, generate will still look fluent — and be wrong. Fix retrieval first.

Two timescales

You pay embedding cost when you index (and when text changes). At question time you mostly embed the question and run chat once (or a few times).

RAG and the agent you already built

RAG is not a new loop. It is a source of facts for Act / Observe — or simply the body of one careful chat call.

Same document-agent story. Tools are optional wrapping.

When things go wrong

Words to keep

See it in Code

The Code panel keeps a tiny handbook in memory, retrieves the closest passages for a laptop-reset question, then asks the chat model to answer from them. Check the Example console under the source.