Embeddings Strategy for ERP Knowledge Bases

Retrieval quality for an ERP knowledge base lives or dies on decisions made long before any query hits a vector index: how the source docs are chunked, which embedding model represents them, and whether search combines keyword and semantic signals or relies on vectors alone. Get these wrong and every downstream RAG answer inherits the error, no matter how good the generation model is.

Chunking strategy for ERP documentation

ERP help content and internal documentation is structurally different from generic prose — it's full of numbered procedures, screen field references, and tables that lose meaning if split mid-table. A naive fixed-size chunker (e.g., 512 tokens with 50-token overlap) will happily cut a procedure in half between step 3 and step 4, and the retrieved chunk becomes useless on its own. Chunk on document structure first — heading boundaries, procedure blocks, table boundaries — and only fall back to fixed-size splitting within a section that's still too long. Keep the parent heading path (e.g., "AP > Bill Entry > Approval Workflow") attached as metadata on every chunk, and prepend it to the chunk text before embedding, because a chunk that just says "select the Approve checkbox and click Save" is meaningless without knowing which screen it's on.

Target chunk size is a tradeoff, not a fixed number: too small and you lose context needed to answer the question; too large and irrelevant text dilutes the embedding and burns context window at generation time. For procedural ERP docs, 200-400 tokens per chunk with structural boundaries tends to outperform arbitrary 512-token windows, because most individual procedures are naturally that length.

Choosing an embedding model

For most ERP knowledge base use cases, a hosted embedding model (OpenAI's text-embedding-3-large, Voyage AI's models, or Cohere embed-v3) is the pragmatic default — you're optimizing for retrieval quality and low operational overhead, not for keeping data fully on-prem, though that constraint does push some regulated deployments toward a self-hosted model like BGE or E5 run locally. Whatever you pick, don't mix embedding models within one index — cosine similarity between vectors from two different models is meaningless — and re-embed the full corpus if you ever change models, rather than only embedding new documents going forward.

Vector DB choice and hybrid search

For an ERP knowledge base in the tens of thousands to low millions of chunks, pgvector alongside your existing Postgres is usually the right call — one less system to operate, and you get transactional consistency between the source-of-truth tables and the index. Dedicated vector databases (Pinecone, Qdrant, Weaviate) earn their keep at larger scale or when you need features like multi-tenant namespace isolation out of the box, but they're a second system to keep in sync with your source content, which is its own maintenance burden.

Pure vector search underperforms on exact-match queries — a user searching for an error code like "PX.20.10" or a specific field name wants that literal string to rank first, and embedding similarity alone won't reliably surface it. Hybrid search — combining a keyword index (Postgres full-text search or BM25) with vector similarity, merged with reciprocal rank fusion — consistently beats either approach alone for this kind of technical content.

WITH semantic AS (
  SELECT id, content, 1 - (embedding <=> $1::vector) AS score
  FROM kb_chunks
  ORDER BY embedding <=> $1::vector
  LIMIT 40
),
keyword AS (
  SELECT id, content, ts_rank(tsv, plainto_tsquery('english', $2)) AS score
  FROM kb_chunks
  WHERE tsv @@ plainto_tsquery('english', $2)
  ORDER BY score DESC
  LIMIT 40
)
SELECT id, content,
  COALESCE(1.0 / (60 + s.rank), 0) + COALESCE(1.0 / (60 + k.rank), 0) AS rrf_score
FROM (SELECT *, ROW_NUMBER() OVER (ORDER BY score DESC) AS rank FROM semantic) s
FULL OUTER JOIN (SELECT *, ROW_NUMBER() OVER (ORDER BY score DESC) AS rank FROM keyword) k
  USING (id)
ORDER BY rrf_score DESC
LIMIT 10;

Re-ranking the candidates

Retrieve a wider candidate set (30-50 chunks) from the hybrid search step, then re-rank with a cross-encoder (Cohere Rerank, or a self-hosted model) before handing the top 5-8 to the generation model. Cross-encoders score the query and chunk together rather than comparing pre-computed vectors, which makes them slower but meaningfully more accurate at judging actual relevance — worth the extra latency at the retrieval stage since it happens once per query, not once per generated token.

Staleness and re-indexing when source docs change

An ERP knowledge base tied to product documentation goes stale the moment a screen changes in the next release, and a stale chunk that ranks highly is worse than no answer at all because it actively misleads. Treat re-indexing as a triggered pipeline, not a cron job that runs blind: hook into whatever publishes the source docs (a CMS webhook, a git push to a docs repo) and re-embed only the changed documents, tracking a content hash per source doc so unchanged sections are never needlessly re-embedded. For docs where staleness risk is high, surface the source doc's last-updated date alongside the answer so users can judge freshness themselves.

Wrapping up

None of the individual pieces here are exotic — structure-aware chunking, a mainstream embedding model, hybrid search, a cross-encoder re-rank step, and a triggered re-indexing pipeline. The quality gap between a mediocre and a genuinely useful ERP knowledge base search almost always traces back to one of these being skipped, not to the choice of generation model sitting on top.