Choosing an embedding model for RAG: a checklist that goes beyond the leaderboard

What the benchmarks tell you

The Massive Text Embedding Benchmark, MTEB, evaluates embedding models across many tasks, such as retrieval, classification, clustering and semantic similarity, over dozens of datasets and many languages. Its central finding when published was that no single method dominated across all tasks¹. Its multilingual successor, MMTEB, presented at ICLR 2025, expanded to over 500 evaluation tasks across hundreds of languages, including long-document and code retrieval². Notably, its authors reported that the best-performing publicly available model at the time was a relatively small multilingual model of around 560 million parameters².

Two lessons follow. Model size isn't the same as fitness for your task. And an average score across hundreds of tasks may hide poor performance on the one task you care about.

The checklist

1. Filter by the task that matters. For RAG, look at retrieval scores, not the overall average. If you search code or long documents, look at those categories specifically.
2. Check your languages. For Gulf and multilingual products, test Arabic and mixed-language queries explicitly. Many models are strongest in English, and a good average can hide weak Arabic retrieval.
3. Check domain fit. Clinical, legal and financial language behave differently from web text. Test with real domain documents; if results are weak, hybrid keyword and vector search often helps more than switching models, as in hybrid search for RAG.
4. Measure on your own data. Build a labelled set of queries and the documents that should be retrieved for them, and measure recall and ranking quality for each candidate. The open-source mteb package can help run standard tasks alongside your own⁴, and the wider method is in evaluation sets that predict production.
5. Consider dimensions and cost. Bigger vectors cost more to store and search. Models trained with Matryoshka Representation Learning let you truncate embeddings to smaller sizes with limited quality loss; the original paper reported up to 14 times smaller embeddings at the same accuracy on an image classification benchmark³. Test the trade-off on your own retrieval set.
6. Consider where it runs. Hosted embedding APIs send your documents to a third party. For regulated or residency-bound data, a self-hosted open model may be necessary, as discussed in Saudi PDPL and cross-border AI and private LLMs on-prem.
7. Check licence and longevity. If a provider deprecates a model, you must re-embed everything. Prefer models you can keep running, or plan for migration.
8. Check context length. Some models truncate long inputs silently. Make sure your chunk sizes fit the model's limit.

Plan for re-embedding from day one

Whatever you choose, you will change it eventually: a better model appears, a provider retires one, or your content shifts language or domain. Design for that:

• Store the model name and version with every vector, and never mix vectors from different models in one index.
• Keep source text and chunking reproducible, so you can re-embed without re-ingesting.
• Build new indexes alongside old ones, compare retrieval quality on your evaluation set, and switch traffic only when the new one wins.
• Budget the cost. Re-embedding a large corpus takes real compute and time; estimate it before you need it.

Where embeddings sit in the bigger picture

Embeddings are one part of retrieval. Reranking, hybrid search, metadata filters and good chunking often matter as much. Where your vectors live is its own decision, covered in pgvector or a vector database. And for some questions, a knowledge graph or GraphRAG beats any embedding, as in GraphRAG vs vector RAG. The most reliable systems treat the embedding model as a swappable, versioned component, chosen with evidence and re-evaluated regularly.

How I can help

I design and evaluate retrieval systems for AI products, including embedding selection, hybrid search and multilingual retrieval for Gulf markets. See my AI architecture work and the RAG to production checklist, or analyse your challenge and bring the result to a call.