Why I built it
By early 2025 I'd read maybe forty blog posts about retrieval-augmented generation and watched three workshops, and I could still only describe it in hand-waves. I decided I'd understand RAG the only way I've ever properly understood anything: by building a small, useless version of it from the ground up, and then making it slowly less useless.
The brief I gave myself was deliberately small. Ingest a folder of PDFs and markdown. Let me ask questions in a chat UI. Always cite which document each answer came from. If you can't ground the answer in a source, say so. No agents, no tool use, no clever stuff — just retrieval and generation done well.
The architecture
The shape ended up looking like every RAG diagram you've ever seen, which I take as evidence I built it right.
┌─────────┐ ┌───────────┐ ┌────────────┐ ┌──────────┐
│ files │ → │ chunker │ → │ embeddings │ → │ Chroma │
└─────────┘ └───────────┘ └────────────┘ └────┬─────┘
│
┌──────────┐ ┌─────────┐ ┌────────┐ ┌─────────▼────────┐
│ user q │ → │ rewrite │ → │ search │ ← │ hybrid: bm25+vec │
└──────────┘ └─────────┘ └───┬────┘ └──────────────────┘
│
┌──────▼──────┐ ┌──────────┐
│ rerank │ → │ LLM │ → cited answer
└─────────────┘ └──────────┘
The interesting parts — the parts where it stopped being a tutorial and started being a real system — were the three boxes in the middle: chunking, hybrid search, and reranking.
Three things that broke, in order
1. Chunking by tokens, not by meaning
My first version split documents into 512-token windows with 50-token overlap. It looked fine in tests and produced absolute garbage in practice. Half the retrieved chunks ended mid-sentence; the LLM would dutifully complete the thought in whatever direction it pleased.
Switched to a recursive splitter that prefers semantic boundaries — paragraphs, then sentences, then tokens as a last resort. Quality jumped immediately. The token-count constraint became a ceiling, not a target.
2. Vector search alone misses exact-match queries
The query "what is the X-1.2 invoice cap" returned nothing useful from pure vector search, because "X-1.2" embedded into roughly the same space as every other product code. Lexical search would have nailed this in milliseconds.
Added BM25 as a parallel retriever, fused with reciprocal rank fusion. Vector search still does the heavy lifting on conceptual queries; BM25 catches exact identifiers and rare terms. Recall@5 went from 0.71 to 0.89 on my eval set.
3. The top-1 chunk was rarely the best one
Retrieval would surface 20 plausible candidates, and the LLM would confidently pick the wrong one if I just slammed them into the prompt ordered by similarity. Reranking turned out to be the highest-leverage change I made.
Added a cross-encoder reranker (BGE-reranker-base) over the top-20 candidates and kept only the top-5 for the prompt. Answers got sharper and the LLM stopped quietly mixing claims from different sources.
What I'd do differently
- Build the eval set first. I built it third. By then I'd already absorbed three rounds of vibes-based feedback that I couldn't undo.
- Use a real document parser for PDFs from day one. PyPDF's output looked clean and was secretly full of header/footer noise that poisoned my chunks.
- Track per-document recall, not just aggregate. One PDF in my corpus had a wonky structure and tanked the average; I almost rebuilt the whole pipeline before I caught it.
What it taught me
RAG is mostly an information retrieval problem with an LLM at the end, not a prompting problem with a search engine at the start. The model can only be as good as what you put in front of it. I came in thinking the prompt was the hard part; I left convinced the chunker and the reranker are.