RAG Reranking Explained: ColBERT, Cross-Encoders, and the LLM Rerank Stack

Vector search gives you 100 plausible documents. Reranking is the line between a RAG demo that wows in a notebook and a system that doesn't hallucinate at 2am. We've shipped retrieval pipelines where adding a reranker moved end-to-end answer accuracy from 71% to 89% on internal eval sets — without touching the LLM, the chunking, or the embedding model.

This post is the tactical version of rag reranking: when to add it, which reranker to pick, what it costs in latency and dollars, and how the two-stage and three-stage stacks actually look in production. If you're still on pure top-k vector search, you're leaving 15-25 points of answer quality on the floor.

Why Retrieval-Only RAG Hits a Recall Ceiling

Dense vector search is fast and cheap but lossy. A 1024-dim embedding has to compress an entire document chunk into a single point in space. That compression collapses subtle distinctions: "refund within 30 days"and "cannot be refunded after 30 days" sit within a cosine similarity of about 0.04 of each other on most embedding models. Bi-encoders simply cannot tell them apart at the top of the ranking.

The textbook fix is to crank top-k. Pull 50, pull 100, pull 200. That works until you remember the LLM is paying per token, and stuffing 100 chunks into a context window introduces a second problem: the lost-in-the-middle effect. Long-context models attend disproportionately to the start and end of the prompt, so doc #47 may as well not exist. Recall went up; precision at the LLM stage cratered.

Reranking solves the asymmetry. You let the cheap retriever cast a wide net (k=100 or k=200), then run a slower, smarter model over those candidates to push the actually-relevant ones to the top. The LLM only sees the top 5 or 10 after rerank. This is the same pattern we covered in our RAG 2025 production guide — retrieval is a funnel, not a single shot.

The Recall vs Precision Tradeoff, Quantified

Those numbers are from a 50K-chunk internal docs corpus. Your mileage varies, but the shape doesn't. Pure vector search at low k is precise but misses. Pure vector at high k recalls everything but drowns the LLM. Add a reranker and you keep the recall of the wide net with the precision of a narrow one.

Cross-Encoder Rerankers: BGE, Cohere Rerank, Voyage

The default move in 2026 is a cross-encoder reranker. Unlike a bi-encoder (which encodes query and document separately and compares vectors), a cross-encoder takes the query and document concatenatedas a single input and outputs a relevance score. The model gets to attend across both texts simultaneously, which is why it's dramatically more accurate — and dramatically slower per pair.

The big three production options:

• BGE-reranker-v2-m3 — open-source, BAAI's flagship. Multilingual, 568M params, self-host on a single A10G. Roughly 80ms per batch of 50 query-doc pairs on a single GPU. Quality is genuinely competitive with Cohere on English.
• Cohere Rerank 3.5 — closed API, $2 per 1K searches. Each "search" is one query against up to 100 documents. P50 latency around 220ms. Best for teams without GPU infra who want plug-and-play quality.
• Voyage rerank-2.5 — closed API, similar pricing, slightly better on code and technical docs in our tests. Multilingual support is solid.

We benchmarked these on a customer-support knowledge base of 80K chunks. NDCG@10 numbers below. The baseline is BM25 + dense hybrid retrieval at k=50 with no rerank.

The gap between self-hosted BGE and the closed APIs is smaller than vendors want you to think. If you have GPU infra and traffic above ~50K queries/day, BGE almost always wins on unit economics. Below that, Cohere's $2/1K is cheaper than paying for an idle A10G. We dig into hybrid retrieval setup in detail in our hybrid search production guide.

ColBERT and Late-Interaction Reranking

ColBERT reranking sits in a strange middle space. It's not quite a bi-encoder (which would be too lossy) and not quite a cross-encoder (which would be too slow at scale). The trick is late interaction: instead of compressing a document into one vector, ColBERT keeps a vector per token, and the query-document similarity is computed by summing the max-similarity match for each query token across all document tokens (MaxSim).

That sounds like a small change but it's a big deal. You get cross-encoder-grade ranking quality with bi-encoder-grade latency, because the document vectors are precomputed and indexed. The catch: storage. A ColBERT index is 50-100x larger than a dense single-vector index for the same corpus. For 1M chunks at 512 tokens average, you're looking at 200-400GB of vector storage before compression.

ColBERTv2 introduced residual compression that brings storage down to roughly 5-10x a dense index. PLAID (the optimized retrieval engine that ships with ColBERTv2) makes it genuinely fast. JaColBERT, Answerai's ColBERTv2 fork, and Stanford's original implementations are all production-grade in 2026.

When ColBERT Beats Cross-Encoders

Cross-encoder rerankers don't scale linearly. If you want to rerank 200 candidates instead of 50, you pay 4x the latency. ColBERT's reranking cost is sub-linear in candidate count because the heavy lifting is precomputed. We've seen ColBERT reranking 500 candidates in the same wall-clock time a cross-encoder reranks 50.

Three concrete scenarios where ColBERT wins:

• Long-tail recall problems where you need k=200+ before the right answer shows up — regulatory docs, legal discovery, scientific literature.
• Multi-vector queries like decomposed sub-questions, where you want fine-grained token matching rather than averaged semantics.
• Low-latency budgets where you can't afford 200ms of cross-encoder time but storage isn't the bottleneck.

Where ColBERT loses: small corpuses (under 100K chunks where the storage overhead isn't worth it), domain-specific jargon that needs explicit fine-tuning, and teams who don't want to manage a second index alongside their dense vector store.

LLM-as-Reranker: When GPT/Claude Beats Cross-Encoders

The newest tier in the reranking llm stack is using the LLM itself as the reranker. You feed the query and the candidate documents (often 20-50) into a single prompt and ask the model to return a ranked list of doc IDs. Done well, this beats every cross-encoder on hard reasoning tasks because the model can actually understand why a document is relevant, not just lexical or semantic surface match.

Three flavors of LLM reranking that show up in production:

1. Listwise (RankGPT-style) — the LLM sees all candidates at once and outputs a permutation. Most accurate, most expensive, hardest to make deterministic.
2. Pointwise — the LLM scores each doc independently on a 1-10 relevance scale. Easier to parallelize, weaker because the model has no comparative signal.
3. Pairwise tournament — the LLM compares docs in pairs, you assemble the ranking from win counts. Reliable but quadratic in cost.

On our internal eval, GPT-4.1-mini doing listwise reranking over 20 candidates beat Cohere Rerank 3.5 by 4-6 NDCG points on questions that required reasoning across multiple snippets. It also cost 30x more and added 800ms of latency. That is not a free lunch.

The pragmatic move: use LLM reranking as a third stage on top of a cross-encoder, not as a replacement. Let BGE or Cohere narrow 100 candidates down to 20, then let Claude Haiku do listwise rerank on those 20 if the query needs deep reasoning. We covered some of the query-side prep in our query augmentation guide.

Latency, Cost, and Accuracy Benchmarks Together

Every reranking choice is a three-axis tradeoff: accuracy, latency, cost. Pick two. Here's what real stacks look like for three common use cases.

The voice agent row is the one teams forget. If your latency budget is sub-250ms end-to-end, you literally cannot afford a cross-encoder. You either ship without rerank or you precompute reranked answers for top intents. Don't pretend you can have it all.

Production Rerank Stack: Two-Stage and Three-Stage Patterns

Here's how we actually wire this together in production. We treat reranking as a strict pipeline with well-defined stage boundaries, because that's the only way to monitor and tune each stage independently.

Two-Stage (the 80% case)

Stage 1 is hybrid retrieval: BM25 + dense vector search, reciprocal rank fusion on the merged result. Pull k=50. Stage 2 is a cross-encoder reranker (BGE or Cohere) that scores all 50 and returns top 8-10 to the LLM.

Operational details that matter:

• Score thresholds, not fixed k. Pass docs to the LLM only if their rerank score exceeds a minimum (e.g., 0.35 for BGE). On easy queries you ship 3 docs, on hard queries you ship 10.
• Cache aggressively. Reranker outputs are pure functions of (query, doc set). Cache on the hash of the candidate IDs. We see 20-30% cache hit rates on customer support workloads.
• Batch within a request. Send all 50 query-doc pairs in one batch call. Per-pair latency drops from 8ms to 2.8ms at batch size 50 on BGE.

Three-Stage (the hard-mode case)

Stage 1: hybrid retrieval, k=200. Stage 2: cross-encoder rerank (BGE) down to top 25. Stage 3: LLM listwise rerank (Claude Haiku) down to top 6 with reasoning. This stack runs around 1.2-1.5s total but the quality gains on complex multi-hop questions are real.

The trap: don't add stage 3 universally. Route to it only when the query passes a complexity classifier — multi-hop questions, comparative questions, questions with negation. A small fine-tuned classifier (DeBERTa-v3-small, 100M params) routes 15-20% of queries to the three-stage path and keeps the rest on the cheap two-stage path. Average cost stays close to two-stage; accuracy on hard queries matches three-stage.

What to Monitor

• Stage-wise recall. Recall@k at stage 1 sets a ceiling. If retrieve recall is 0.85, no reranker fixes it.
• Rerank score distribution. Drift in mean rerank score per day signals corpus or query distribution shift.
• Top-1 churn. How often does the post-rerank top doc differ from the pre-rerank top doc? Below 30% means your reranker isn't earning its latency.
• Cache hit rate. Below 10% means you're burning money. Below 5% means your query distribution is too long-tail for caching to help; reconsider cost.

We've written more on the broader category in our deep-dive on reranking strategies for RAG, which goes into fine-tuning rerankers on your own labeled data.

The Practical Decision Tree

If you're starting from scratch:

1. Get hybrid retrieval working first. Pure vector is leaving 10+ NDCG points on the table.
2. Add Cohere Rerank as a one-line API call. Measure the lift. If it's less than 5 NDCG points, your retriever or chunks are the actual bottleneck — fix those first.
3. Once Cohere is winning, evaluate self-hosted BGE if your volume justifies GPU costs.
4. Add LLM listwise reranking only on the slice of queries that demonstrably need reasoning.
5. Consider ColBERT if your corpus is large, your latency budget is tight, and you need k > 200.

The mistake we see repeatedly: teams jump straight to LLM reranking because it's the shiny new thing, skip cross-encoders entirely, and end up with a $60/1K-query stack that's only 4 NDCG points better than a $2/1K stack would have been. Reranking is a stack, not a switch. Build it in order.

Key Takeaways

• Reranking is non-optional for any RAG system above demo quality. Expect 25-35% NDCG@10 lift over hybrid-only retrieval.
• Cross-encoders (BGE, Cohere, Voyage) are the default. Pick based on volume and infra preference.
• ColBERT wins when you need to rerank 200+ candidates and have storage headroom.
• LLM listwise reranking is the ceiling on quality but costs 15-30x more — route to it selectively.
• Two-stage covers 80% of cases. Three-stage with query routing handles the rest.
• Monitor stage-wise recall and top-1 churn — without those metrics you're flying blind.