Build a RAG App From Scratch

Why RAG exists

Large language models are trained on public data up to a cutoff date. They don't know your company handbook, your product docs, or yesterday's meeting notes. Ask them anyway and they'll often guess — confidently. (See why models hallucinate.)

You could paste all your documents into every prompt, but that quickly becomes slow, expensive, and eventually exceeds the context window. Retrieval-augmented generation (RAG) solves this: for each question, find only the few most relevant passages, and give those to the model along with the question.

How RAG works

RAG has two phases:

The core trick is semantic search. An embedding model turns any text into a vector so that texts with similar meaning have vectors pointing in similar directions. "How many vacation days do I get?" will land close to a passage about "annual paid leave allowance" — even though they share almost no words.

Setup

You'll need Python 3.9+ and two libraries. We'll use sentence-transformers to run a small, free embedding model locally, and numpy for the math.

pip install sentence-transformers numpy

Create a folder called docs/ and put a few .txt files in it — policies, notes, FAQs, anything you'd like to ask questions about.

Step 1: Load documents

from pathlib import Path

def load_documents(folder="docs"):
    docs = []
    for path in sorted(Path(folder).glob("*.txt")):
        docs.append({"source": path.name, "text": path.read_text(encoding="utf-8")})
    return docs

documents = load_documents()
print(f"Loaded {len(documents)} documents")

In real projects this step also handles PDFs, web pages and databases. Clean text in means good answers out — parsing quality matters more than most people expect.

Step 2: Chunk them

We split each document into overlapping passages. Why not embed whole documents? A single vector for a 30-page document blurs many topics together, and retrieving the whole thing wastes context. Smaller chunks give precise matches.

def chunk_text(text, chunk_size=800, overlap=150):
    """Split text into ~chunk_size-character pieces that overlap,
    so a sentence cut at a boundary still appears whole in one chunk."""
    chunks, start = [], 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end].strip())
        if end >= len(text):
            break
        start = end - overlap
    return [c for c in chunks if c]

chunks = []
for doc in documents:
    for i, piece in enumerate(chunk_text(doc["text"])):
        chunks.append({"source": doc["source"], "id": i, "text": piece})

print(f"Created {len(chunks)} chunks")

Step 3: Embed the chunks

Now we turn every chunk into a vector. all-MiniLM-L6-v2 is a small, fast open-source embedding model that runs fine on a laptop CPU and produces 384-dimensional vectors.

import numpy as np
from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer("all-MiniLM-L6-v2")

chunk_vectors = embedder.encode(
    [c["text"] for c in chunks],
    normalize_embeddings=True,   # unit length, so dot product = cosine similarity
    show_progress_bar=True,
)
print(chunk_vectors.shape)  # (number_of_chunks, 384)

That NumPy array is our vector index. For a few thousand chunks, it's all you need. For millions, you'd store these vectors in a vector database (pgvector, Qdrant, Pinecone, etc.) — the concept is identical.

Step 4: Retrieve relevant chunks

To answer a question, we embed it with the same model and find the chunks whose vectors are most similar. Because the vectors are normalized, a dot product gives the cosine similarity: 1.0 means identical direction, around 0 means unrelated.

def retrieve(question, k=4):
    q = embedder.encode([question], normalize_embeddings=True)[0]
    scores = chunk_vectors @ q                  # similarity to every chunk
    top = np.argsort(scores)[::-1][:k]          # indices of the k best
    return [(chunks[i], float(scores[i])) for i in top]

for chunk, score in retrieve("How many vacation days do employees get?"):
    print(f"{score:.3f}  {chunk['source']}  {chunk['text'][:80]}...")

Run this and inspect the results before involving an LLM at all. If retrieval returns the wrong passages, no model can give the right answer. Most RAG quality problems are retrieval problems.

Step 5: Generate a grounded answer

Finally, we build a prompt that contains the retrieved passages and clear instructions, then send it to an LLM.

def build_prompt(question, retrieved):
    context = "\n\n".join(
        f"[{i + 1}] (source: {c['source']})\n{c['text']}"
        for i, (c, _score) in enumerate(retrieved)
    )
    return f"""Answer the question using ONLY the context below.
If the answer is not in the context, say "I don't know based on the provided documents."
Cite the passages you used with their numbers, like [1] or [2].

Context:
{context}

Question: {question}
Answer:"""


def call_llm(prompt):
    """Send the prompt to the LLM of your choice and return its text reply.
    Use your provider's official SDK (Anthropic, OpenAI, Google, ...)
    or a local model served by Ollama."""
    raise NotImplementedError("Connect your LLM provider here")


def answer(question):
    retrieved = retrieve(question)
    return call_llm(build_prompt(question, retrieved))

Three details in that prompt do a lot of work:

• "Using ONLY the context" keeps the model grounded in your documents rather than its training data.
• An explicit way to say "I don't know" dramatically reduces made-up answers.
• Numbered citations let users verify answers — and let you debug which chunk caused a bad one.

That's a complete RAG system in under 70 lines. Everything production RAG frameworks do is an elaboration of these five steps.

Making it production-grade

Once the basics work, these are the upgrades that matter most, roughly in order of impact:

1. Build an evaluation set

Write 30–100 real questions with known correct answers and the passages that contain them. Measure two things separately: retrieval (did the right passage appear in the top-k?) and generation (was the final answer correct and faithful to the context?). Re-run it after every change.

2. Hybrid search

Embeddings capture meaning but can miss exact terms like product codes, error numbers or names. Combine vector search with keyword search (BM25) and merge the results — a common technique is reciprocal rank fusion.

3. Reranking

Retrieve a wider set (say, the top 20), then use a reranker (cross-encoder) model to re-score each passage against the question and keep the best 4–5. Rerankers are slower but much more precise than embedding similarity alone.

4. Smarter chunking and metadata

Split on document structure, prepend each chunk with its document title and section heading, and store metadata (date, department, access level) so you can filter results — for example, only showing documents the user is allowed to see.

5. Query rewriting

In a chat, follow-ups like "what about part-timers?" don't make sense alone. Use the LLM to rewrite the question into a standalone query, using the conversation history, before retrieving.

6. Keep the index fresh

Re-embed documents when they change, remove deleted ones, and record which embedding model created each vector — switching models means re-embedding everything.

Key takeaways

• RAG = retrieve relevant passages, then let the LLM answer from them.
• Pipeline: load → chunk → embed → retrieve → generate.
• Embeddings enable search by meaning, not just keywords.
• Test retrieval on its own first — most failures happen there.
• Prompts should restrict answers to the context, allow "I don't know", and require citations.
• Evals, hybrid search and reranking are the biggest upgrades for production.