First-pass retrieval (keyword, vector, or hybrid) is built for speed over a large corpus. It is good at getting the right answer somewhere in the top 20–100, and less good at putting the single best paragraph in slot one. Reranking takes that shortlist and scores each candidate more carefully against the question so the LLM sees the best few chunks.
This article stays introductory: what reranking is, why cross-encoders help, and how to choose a sensible candidate count.
The two-stage pattern
- Retrieve broadly with a cheap method (BM25, embeddings, or hybrid) → maybe 50 chunks.
- Rerank narrowly with a stronger model → keep the top 3–8 for the prompt.
You pay a little extra latency so the generator wastes fewer tokens on weak context.
Bi-encoder vs cross-encoder (plain language)
Bi-encoder (typical vector search): embed the query once, compare to vectors you stored earlier. Fast, scalable, a bit approximate.
Cross-encoder (typical reranker): feed the pair (question, chunk) into one model and get a relevance score. The model can look at both texts together, so ranking is usually sharper—but you cannot precompute every possible pair for a million documents. That is why it only runs on a shortlist.
Managed options exist too (for example semantic rankers in cloud search products). The idea is the same: expensive scoring on a small set.
How many candidates should you rerank?
Rules of thumb for beginners:
- If N is too small (for example 5), the right chunk may never enter the reranker.
- If N is too large (for example 500), latency and cost jump with little gain.
- A common starting point: retrieve 20–100, keep 3–8 after rerank.
Tune with your eval set: watch recall before vs after rerank, plus p95 latency.
Minimal usage sketch
# Pseudocode
candidates = hybrid_search(question, top_n=50)
scored = [
(chunk, cross_encoder.score(question, chunk.text))
for chunk in candidates
]
top = sorted(scored, key=lambda x: x[1], reverse=True)[:5]
context = "\n\n".join(c.text for c, _ in top)
# then call the LLM with context + question
Batch scoring when the API or library supports it; scoring one-by-one in a loop is the slow path.
When reranking helps most
- Your first-pass recall@50 is decent, but answer quality is unstable.
- Hybrid search returns mixed keyword and semantic hits that need a fair final order.
- You want fewer, better chunks in the prompt to cut cost and confusion.
When it helps less:
- The right document is not in the index at all (fix chunking or coverage first).
- Queries are pure exact-ID lookups and BM25 already ranks them perfectly.
Beginner checklist
- Keep a labeled question set; measure before/after.
- Rerank only a shortlist, not the whole corpus.
- Cap output to a handful of chunks for the LLM.
- Log scores and chosen chunk IDs for debugging.
- Watch latency budgets—rerank sits on the user-facing path.
Reranking is a small stage with an outsized effect: let a fast retriever cast a wide net, then let a careful scorer decide what the model is allowed to read.
Related reading
- Hybrid Search in RAG: Combining BM25 Keyword and Vector Retrieval (companion draft in this series): a strong first-pass list makes reranking more effective.
- Building a RAG Pipeline from Scratch: Embeddings, Vector Store, and Generation: where retrieval and prompting fit together end to end.
About the author
I write about Applied AI, MCP, Azure AI Search, and RAG. Follow me on Dev.to and Hashnode for more.



Top comments (0)