AI Agent for Internal Knowledge Base

An AI agent that answers questions from a knowledge base is a retrieval system wearing a chat interface. The LLM part — reading retrieved passages and composing an answer — is the easy 20%. The 80% is getting the right passages into the context window in the first place: chunking documents sensibly, embedding them well, retrieving with enough precision that the model isn't drowning in irrelevant text, and being honest when nothing relevant exists.

Chunking is where quality is won or lost

Fixed-size chunking (split every 500 tokens) is the easiest thing to ship and the first thing that quietly caps your answer quality. It slices sentences in half, separates a heading from the paragraph it introduces, and splits a table from its caption. The fix that pays for itself immediately is structure-aware chunking: split on markdown headings, HTML sections, or paragraph boundaries first, and only fall back to a token-count split for oversized sections. Keep a healthy overlap (10-15% of chunk length) between adjacent chunks so a sentence that references "the previous paragraph" doesn't lose its antecedent. For anything with real hierarchy — a manual with sections and subsections — store the heading path alongside each chunk ("Setup > Prerequisites > Database") and prepend it to the chunk text before embedding; it gives the embedding model context it wouldn't otherwise have and gives the LLM a citation-worthy label later.

Embeddings alone are not enough — go hybrid

Pure vector similarity search misses exact matches: a part number, an error code, an acronym. Embedding models are trained for semantic similarity, and "ERR_4092" doesn't have a semantically similar neighbor, it just needs to be found verbatim. Production retrieval that actually holds up combines dense vector search with a keyword/BM25 index (Postgres full-text search, Elasticsearch, or a vector DB with built-in hybrid search like Pinecone or Weaviate both support this now) and merges the two result sets, typically with reciprocal rank fusion. This single change fixes more retrieval complaints than swapping embedding models does.

def reciprocal_rank_fusion(vector_results, keyword_results, k=60):
    scores = {}
    for rank, doc_id in enumerate(vector_results):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    for rank, doc_id in enumerate(keyword_results):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

def retrieve(query, top_k=8):
    vector_hits = vector_index.search(embed(query), top_k=20)
    keyword_hits = bm25_index.search(query, top_k=20)
    fused = reciprocal_rank_fusion(
        [h.id for h in vector_hits],
        [h.id for h in keyword_hits],
    )
    return [chunk_store[doc_id] for doc_id in fused[:top_k]]

Rerank before the context window, not after

Retrieving 20 candidates and stuffing all 20 into the prompt is how context windows fill up with noise and answer quality drops — models get measurably worse at using information buried in the middle of a long context ("lost in the middle"). Retrieve broad (top 20-30), then rerank with a cross-encoder (Cohere Rerank, or a local model) that scores query-passage pairs directly rather than via cosine similarity, and pass only the top 5-8 reranked chunks to the LLM. Cross-encoders are slower per-pair but you're only running them on the retrieval shortlist, not the whole corpus, so the added latency is small relative to the quality gain.

Knowing when there's no good answer

The default failure mode of a RAG agent isn't returning the wrong document — it's confidently answering from documents that don't actually address the question, because retrieval always returns something, even when nothing in the corpus is relevant. Set a similarity/rerank-score floor below which you tell the model explicitly "no relevant context was found, say so rather than guessing," and instruct the model in the system prompt to answer only from the provided context and to say when it can't. This is a prompt instruction that's worth having even though prompt instructions generally aren't security boundaries — here the downside of the model ignoring it is a bad answer, not a compromised system, and instruction-following on "don't answer outside the given context" is one of the things current frontier models are reasonably good at when asked directly.

Evaluating retrieval quality separately from generation quality

Bad final answers can come from two different failures — retrieval brought back the wrong chunks, or generation misused the right chunks — and conflating them makes debugging slow. Keep a small hand-labeled eval set (50-200 query/expected-source pairs is enough to start) and measure retrieval recall (is the right chunk in the top-k?) independently from answer faithfulness (does the generated answer actually follow from what was retrieved?). Run both on every meaningful change to chunking, embedding model, or reranker before shipping it — retrieval regressions are easy to introduce and easy to miss without a standing eval.

Wrapping up

Most of the effort in a knowledge-base agent belongs in the retrieval pipeline, not the prompt. Chunk with document structure in mind, combine vector and keyword search, rerank before you fill the context window, and give the model an explicit escape hatch for "not in the corpus." Keep retrieval and generation evaluated separately so a bad answer tells you which half of the system actually broke.