Every RAG or semantic search project has the same first decision buried in it: which model turns your text into vectors. It's easy to default to whatever's most talked-about and move on, but the choice affects cost, latency, retrieval quality, and how much control you have over the model long-term — and the trade-offs are more concrete than "bigger model, better results."
Two families: hosted APIs and open sentence-transformers
OpenAI's text-embedding-3 family and Cohere's embed-v3 are hosted APIs — you send text, get a vector back, pay per token, and never think about GPU infrastructure. Sentence-transformers models (the all-MiniLM-L6-v2, bge-*, e5-* family, mostly built on the sentence-transformers library on top of Hugging Face models) run locally or on infrastructure you control — no per-call cost, no network round-trip, but you own the hosting, batching, and GPU/CPU sizing. The quality gap between the best open models and the hosted APIs has narrowed a lot over the last couple of years; on the MTEB leaderboard, open models like BGE and E5 variants routinely sit near or above OpenAI's offerings on many task categories, so "the API model is just better" isn't a safe assumption anymore — benchmark on your own data.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("BAAI/bge-base-en-v1.5")
docs = ["Acumatica GI performance tuning", "indexing large SQL views"]
embeddings = model.encode(docs, normalize_embeddings=True)
print(embeddings.shape) # (2, 768)from openai import OpenAI
client = OpenAI()
response = client.embeddings.create(
model="text-embedding-3-small",
input=["Acumatica GI performance tuning", "indexing large SQL views"]
)
vector = response.data[0].embedding # length 1536 by defaultDimensionality: bigger vectors aren't free
Embedding dimensionality (384 for MiniLM, 768-1024 for BGE-large/E5-large, 1536 or 3072 for OpenAI's text-embedding-3 models) is a direct trade-off between representational capacity and storage/compute cost. A 1536-dimension vector takes 4x the storage of a 384-dimension one at the same precision, and nearest-neighbor search cost scales with dimensionality too. OpenAI's newer embedding models support "Matryoshka" dimensionality reduction — you can truncate the vector to a smaller size (say 256 dimensions) and retain most of the useful signal, trading a small quality hit for a real reduction in storage and search cost. Not every model supports this; check whether truncation is officially supported before just slicing a vector, since not all embedding spaces are structured to tolerate it gracefully.
Most embedding models are trained so a normalized dot product (cosine similarity) is the intended similarity measure. If your vector database defaults to raw dot product and your vectors aren't normalized to unit length, longer documents can score artificially higher purely from vector magnitude. Normalize at embedding time and use cosine similarity, or normalize once and use dot product — they become equivalent once vectors are unit-length.
Cosine similarity vs dot product in practice
Cosine similarity measures the angle between two vectors, ignoring magnitude — it answers "are these two pieces of text pointing in the same semantic direction," which is usually what you want for retrieval. Raw dot product also factors in vector magnitude, which can matter if your model encodes something meaningful in vector length (some retrieval-tuned models do, deliberately, to represent document importance or specificity). Most vector databases (pgvector, Pinecone, Qdrant, Weaviate) let you pick the distance metric per index — cosine is the safe default unless you have a specific reason and have verified your model's training objective actually uses magnitude meaningfully.
When fine-tuning actually beats an off-the-shelf model
Fine-tuning an embedding model is worth the cost when your domain vocabulary diverges sharply from general web text and you have enough labeled pairs (positive/negative examples, or query-document pairs) to actually move the needle — a few thousand well-curated pairs is a reasonable starting point, fewer than that and you risk overfitting to noise. In practice, most teams get more value from better chunking strategy and a good reranker than from fine-tuning embeddings first — fine-tuning is the last lever I pull, not the first, because it's the one with the highest ongoing maintenance cost (you now own model versioning and retraining, forever).
Vectors from different models (or different dimensionality settings of the same model) are not comparable to each other — cosine similarity between a BGE vector and an OpenAI vector is meaningless. If you switch embedding models, you must re-embed your entire corpus, not just new documents. Plan for this before you pick a model, not after your index has a million vectors in it.
Wrapping up
There's no universally correct embedding model — the real decision tree is: do you want zero infra and pay-per-call pricing (hosted API), or do you want cost control and no per-token bill at the price of running your own inference (sentence-transformers)? Check MTEB or your own domain-specific eval before assuming quality favors either camp, pick a dimensionality that matches your actual storage and latency budget, and treat fine-tuning as an optimization you earn only after chunking, retrieval, and reranking have already been tuned.
Independent software engineer in Nairobi specialising in Acumatica customisations, Laravel backends, and tax fiscalisation integrations across East and Southern Africa.