The first time I built a retrieval-augmented chatbot, I followed a tutorial. Five hundred tokens per chunk, fifty tokens of overlap, embed everything, store it in Chroma, retrieve top-k, stuff it into the prompt. It worked in the way that all tutorials work — beautifully on the sample document, vaguely on anything I cared about.
The first time I deployed it to friends, the bug arrived inside an hour. They asked simple questions and the bot gave answers that were almost-right in a very specific way: the facts were real but the reasoning was wrong, and I could never quite tell where the rot started. It took me three full rebuilds of the pipeline to figure out what was happening.
What was actually wrong
Five-hundred-token windows don't respect sentence boundaries. They certainly don't respect paragraph boundaries. What they do, with enthusiastic consistency, is end in the middle of an idea — and the LLM, faced with a truncated thought, will dutifully complete it in whatever direction it pleases.
I'd retrieved this chunk for a question about refund policy:
"…the customer is entitled to a full refund within fourteen days, except in cases where the product has been opened, in which case the customer is entitled to"
Entitled to what? The chunk ends. The LLM picks. The answer is plausible, internally consistent, and completely made up. And because I'd evaluated the system by asking it questions and reading the answers, I couldn't tell. The hallucination looked like a quotation.
Fix one: chunk by meaning, not by tokens
The smallest thing that helped was switching from a fixed-window splitter to a recursive one that prefers semantic boundaries. Paragraphs first; if a paragraph is too big, fall back to sentences; if a sentence is too big, fall back to tokens as the floor, but never as the goal. The token count became a ceiling, not a target.
This isn't novel — LangChain's RecursiveCharacterTextSplitter
has done it forever — but the lesson generalized. Anywhere in a pipeline
where I was making decisions by counting (tokens, bytes, lines), I asked
myself whether the count was a constraint or a target, and 80% of the time
it was a constraint pretending to be a target.
Fix two: hybrid search
Once the chunks made sense, I noticed another failure pattern. The query "what is the X-1.2 invoice cap" returned nothing useful. Embeddings put "X-1.2" into roughly the same neighbourhood as every other product code in the corpus. A lexical search would have nailed it in milliseconds.
So I ran BM25 in parallel with the vector search and fused the two with reciprocal rank fusion:
# weight-free combiner: each doc's score is the sum of 1/(k + rank)
# across however many retrievers found it
def rrf(rank_lists, k=60):
scores = {}
for ranks in rank_lists:
for r, doc_id in enumerate(ranks):
scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + r)
return sorted(scores, key=scores.get, reverse=True)
Vector search kept its lead on conceptual queries. BM25 caught the rare terms and exact identifiers it had no business missing. Recall@5 went from 0.71 to 0.89 on my eval set, which I'd finally gotten around to building.
Fix three: rerank before you stuff
With both retrievers running, I now had 20 plausible candidates per query. Slamming all 20 into the prompt was expensive and didn't actually work — the LLM would helpfully synthesize an answer that mixed claims from unrelated chunks. The fix turned out to be the highest-leverage change in the whole pipeline.
A cross-encoder reranker over the top-20, keeping only the top-5 for the prompt. BGE-reranker-base is small enough to run locally, and the latency cost was tiny compared to the answer-quality improvement. Once it was in place, the model stopped quietly mixing claims from different sources, and the answers started to feel like they came from one mind instead of three.
The actual lesson
I came into RAG thinking the prompt was the hard part. I left convinced the chunker and the reranker are. RAG is mostly an information retrieval problem with an LLM at the end, not a prompting problem with a search engine at the start. The model can only ever be as good as what you put in front of it.
I also learned to build the eval first, third, or third. By the time I had an eval, I'd already absorbed three rounds of vibes-based feedback I couldn't undo. Now my unbreakable rule is: write thirty cases before writing a line of pipeline code. The pipeline is easy. The cases are where the actual problem hides.