"Attach a vector database to your LLM and you have RAG" is the pitch, and it's the part everyone gets working in an afternoon. It's also the least important part. Retrieval-augmented generation lives or dies on retrieval quality — whether the chunks you hand the model actually contain the answer — and that's the part that takes real iteration. I've built enough of these to have a fairly settled view of where the basic pipeline breaks and which patterns are worth the added complexity.
The basic pipeline
Every RAG system does the same five things: split source documents into chunks, embed each chunk into a vector, store the vectors in an index, retrieve the top-k chunks most similar to the query's embedding, and stuff those chunks into the prompt alongside the user's question for the model to generate from. Optionally a re-ranking step sits between retrieval and generation, re-scoring the initial candidates with a more expensive, more accurate model before truncating to the final set that goes in the prompt. None of this is hard to wire up — LangChain, LlamaIndex, or a dozen vector DB SDKs will get you a working pipeline in under a hundred lines. The hard part is everything that determines whether the retrieved chunks are actually the right ones.
from sentence_transformers import SentenceTransformer
import numpy as np
embedder = SentenceTransformer("all-MiniLM-L6-v2")
def chunk_text(text, size=500, overlap=50):
words = text.split()
chunks = []
for i in range(0, len(words), size - overlap):
chunks.append(" ".join(words[i:i + size]))
return chunks
chunks = chunk_text(source_document)
chunk_vectors = embedder.encode(chunks, normalize_embeddings=True)
def retrieve(query, k=5):
query_vector = embedder.encode([query], normalize_embeddings=True)[0]
scores = chunk_vectors @ query_vector # cosine similarity via dot product
top_k = np.argsort(scores)[::-1][:k]
return [(chunks[i], float(scores[i])) for i in top_k]
results = retrieve("What is the cancellation policy?")Where naive RAG breaks down
Fixed-size chunking is the first thing that goes wrong: a 500-word window has no idea where a sentence, a section, or a table ends, so it routinely splits the sentence that contains the actual answer across two chunks — the retriever finds the chunk with the question restated and misses the chunk with the number. The second problem is the precision/recall tradeoff baked into top-k retrieval: ask for more chunks and you dilute the prompt with irrelevant context that can distract the model or push out the chunk that mattered; ask for fewer and you risk missing the right one entirely. The third is staleness — embeddings are a snapshot of the source document at index time, and if the underlying document changes (a policy update, a price change) and you don't re-embed, the model confidently answers from a chunk that's now wrong, with no signal to anyone that it happened.
When a RAG system gives a wrong answer, the instinct is to blame the LLM and tweak the prompt. Most of the time the model answered correctly given what it was handed — the actual bug is that the retriever fetched the wrong chunks. Always inspect what was retrieved before touching the prompt.
Hybrid search: keyword plus vector
Pure vector similarity is bad at exact-match queries — product SKUs, error codes, proper nouns — because embedding models compress meaning, not surface form, and two dissimilar-looking strings that mean similar things end up close in vector space while an exact code match can end up far apart. Hybrid search runs a traditional keyword index (BM25 is the standard) alongside the vector search and combines the two ranked lists, typically with reciprocal rank fusion, before re-ranking. This catches the cases vector search alone misses — a user searching for an exact error code or a specific model number — without losing the semantic matching that keyword search alone can't do. Most production-grade vector stores (Elasticsearch, Weaviate, Qdrant) now support this natively rather than requiring you to run two separate systems and merge results yourself.
Hierarchical and parent-document retrieval
Small chunks retrieve precisely but lack context; large chunks carry context but retrieve imprecisely and waste prompt space on irrelevant text. Hierarchical (parent-document) retrieval resolves this by embedding small chunks for search but returning their larger parent section — a paragraph or a document's containing chapter — once one of its child chunks is matched. The retriever's similarity search stays sharp because it's matching against small, focused text, but the model gets the surrounding context it needs to actually use the retrieved fact correctly, instead of a fragment missing the qualifier two sentences earlier that changes its meaning.
A cross-encoder re-ranker (e.g. a small BERT-based model scoring query-chunk pairs directly, rather than comparing pre-computed embeddings) is slower per-comparison than vector search but only needs to run on the top 20-50 candidates, not the whole corpus. Adding it after initial retrieval consistently improves precision for a latency cost measured in tens of milliseconds — cheap relative to the LLM call that follows it.
Keeping the index current
Treat the vector index as a derived artifact of the source documents, not as the source of truth itself — the same relationship a search index has to a database. That means a re-embedding pipeline triggered by document changes (a webhook on document update, or a scheduled diff-and-reembed job), not a one-time ingestion script run at project kickoff and forgotten. For systems where staleness has real cost — pricing, policy, compliance documents — track a last-indexed timestamp per document and either exclude or explicitly flag chunks older than a threshold, so a stale answer is at least visible as stale rather than presented with the same confidence as a fresh one.
Wrapping up
The generation half of RAG is close to a solved problem — any current-generation LLM will write a coherent answer from good context. The retrieval half is where the actual engineering work is: chunking strategy, hybrid search to cover exact-match queries vector similarity misses, hierarchical retrieval to balance precision against context, re-ranking to clean up the candidate set, and a real pipeline for keeping the index in sync with source documents. If a RAG system is giving wrong answers, look at what got retrieved before you touch the prompt — that's where the bug almost always is.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.