Vector Search Tuning — A Field Guide

Getting a vector index running is the easy part — every vector database ships a "create index, insert vectors, query" quickstart that works fine on ten thousand rows. Tuning it to hold acceptable recall and latency at ten million rows, under real query load, is the part nobody's quickstart covers. Most of what actually matters comes down to a handful of HNSW parameters, an honest decision about approximate vs. exact search, and whether you're relying on vector similarity alone when you shouldn't be.

HNSW: the index almost everyone is actually running

HNSW (Hierarchical Navigable Small World) is the approximate nearest-neighbor algorithm behind most production vector search today — pgvector, Qdrant, Weaviate, and Pinecone all use it or a close variant. It builds a multi-layer graph where each vector is a node, and search descends from a sparse top layer to a dense bottom layer, following edges toward the query vector. Three parameters control the trade-off between recall, latency, and memory: M (max connections per node — higher M means a denser, more accurate graph and more memory), ef_construction (how thoroughly the graph is built at insert time — higher means a better graph but slower indexing), and ef_search (how many candidates are explored per query — higher means better recall at the cost of query latency).

CREATE INDEX ON documents
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 100);

-- ef_search is set per-query/session, not baked into the index
SET hnsw.ef_search = 80;

SELECT id, content
FROM documents
ORDER BY embedding <=> '[0.01, 0.02, ...]'::vector
LIMIT 10;

M between 12-48 and ef_construction between 100-200 cover most workloads reasonably — pgvector's own defaults (m=16, ef_construction=64) are a sane starting point, not something you need to override on day one. ef_search is the knob to reach for first when tuning an existing index, since it's changeable at query time without a rebuild: start low, measure recall against a labeled query set, and raise it until recall is acceptable, watching the latency cost as you go.

Approximate vs exact nearest neighbor

Exact nearest neighbor (brute-force cosine distance against every vector) guarantees perfect recall but scales linearly with corpus size — fine for tens of thousands of vectors, painful past a few hundred thousand depending on your latency budget and hardware. HNSW and other ANN (approximate nearest neighbor) indexes trade a small, tunable amount of recall for sublinear query time. The trade-off is genuinely tunable, not all-or-nothing: at high ef_search values HNSW recall against a brute-force baseline is often well into the high nineties percent, at the cost of higher latency and memory versus a lower ef_search setting. For most search and RAG use cases, that gap is invisible to end users; for exact-match-critical use cases (deduplication, legal discovery), consider exact search or a hybrid exact-rerank step on the ANN shortlist.

Hybrid search: vector similarity alone misses things

Pure vector search is weak at exact keyword matches — a query for an error code, a product SKU, or a person's name can score poorly on embedding similarity even when a document contains that exact string, because embeddings are tuned for semantic similarity, not lexical overlap. Hybrid search combines a vector search with a traditional keyword/BM25 search (via Postgres full-text search, Elasticsearch, or a vector DB's built-in sparse-vector support) and merges the two ranked lists, typically with Reciprocal Rank Fusion (RRF) rather than trying to normalize and average two incomparable score scales.

def reciprocal_rank_fusion(vector_results, bm25_results, k=60):
    scores = {}
    for rank, doc_id in enumerate(vector_results):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    for rank, doc_id in enumerate(bm25_results):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

RRF sidesteps the score-normalization problem entirely by only using rank position, not raw scores, which is why it's become the default fusion method in most hybrid setups (Qdrant, Weaviate, and Elasticsearch all support it natively now).

Re-ranking: a second, more expensive pass on a short list

A cross-encoder reranker (like Cohere Rerank or an open model such as bge-reranker) scores a query against each candidate document jointly, rather than comparing independently-computed embeddings — much more accurate, much more expensive per comparison, which is why it's only run on the top 20-100 candidates already retrieved by vector or hybrid search, not the whole corpus. The retrieval step's job is to cheaply narrow millions of documents to a short list with decent recall; the reranker's job is to get the ordering of that short list right. Skipping reranking is fine for low-stakes search; for RAG pipelines where the top 3-5 chunks go straight into an LLM prompt, a reranker measurably improves what the model actually sees.

Wrapping up

Vector search tuning in production is mostly about three levers pulled in order: get the HNSW parameters (ef_search especially) into a range that balances recall against your latency budget, add keyword/BM25 hybrid search so exact matches don't fall through the cracks that pure embeddings leave, and layer a reranker on the retrieved candidates when result quality — not just recall — is what's being measured. None of this requires abandoning approximate search for exact search; it requires actually measuring recall against a labeled query set instead of assuming the defaults are fine.