AI Guides · 10 min read

What Is a Reranker? How Cross-Encoders Refine RAG Results

A reranker rescores retrieved candidates, usually with a cross-encoder, after a first-stage search. When that extra step is worth the latency and cost.

A tighter row of blank, document-like cards is drawn toward a bright focal card in the foreground, while more cards float scattered in the dark background, with no text or logos

A reranker is a second-stage model that takes a query and a short list of candidates from a first-stage search, then rescores those items so the most relevant passages move to the top. In retrieval-augmented generation (RAG), reranking narrows broad search hits down to the top chunks sent to the language model.

Search engines and retrieval-augmented generation (RAG) pipelines face a basic tension between speed and ranking accuracy. Scanning millions of passages requires a cheap lookup method, yet cheap similarity scores often put tangential passages ahead of the exact answer. Passage reranking resolves that tension by splitting retrieval into a broad, fast candidate search followed by a slower, more attentive scoring pass over a short list.

How Does Two-Stage Retrieval Work in Search and RAG?

Two-stage retrieval separates fast candidate generation across a full corpus from high-precision scoring on a small subset of results. Information-retrieval systems have used this pattern long before modern language models, and RAG stacks adopt the same division of labor to protect the prompt context window from noisy hits.

A standard two-stage RAG pipeline runs in three sequential steps:

  1. First-stage retrieval (candidate generation): A fast retriever searches the full collection and returns tens to a few hundred candidate chunks. Common first-stage methods include lexical search over BM25-style inverted indexes, dense retrieval using text embeddings and approximate nearest-neighbor (ANN) search in a vector database, or a hybrid combination of both.
  2. Passage reranking: A dedicated reranker evaluates the user query alongside each candidate chunk from the first stage and assigns a fresh relevance score, reordering the shortlist.
  3. Generation: The application keeps only the highest-scoring top-n passages after reranking and places them into the language model prompt alongside the user question.

First-stage retrieval optimizes for recall and low latency across a large index. Passage reranking optimizes for precision at the very top of a short list. Adding a reranker leaves the downstream language model and your underlying system of record unchanged; the reranker simply decides which retrieved chunks earn a spot in the prompt.

Bi-Encoder vs. Cross-Encoder: Why Can’t One Model Do Both?

Bi-encoders and cross-encoders process queries and documents through different computation paths, creating a direct tradeoff between offline indexing and joint token-level comparison. Understanding bi-encoder vs cross-encoder architecture explains why almost no production system runs a cross-encoder over an entire database, and why vector similarity alone sometimes misorders close candidates.

How Bi-Encoders Enable Fast First-Stage Retrieval

A bi-encoder (also called a dual encoder or embedding model) passes the query and each document through two independent encoding passes to produce separate fixed-length vectors. Nils Reimers and Iryna Gurevych formalized a widely cited version of this Siamese architecture for text in Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (EMNLP 2019).

Because document encoding never sees the user query, every chunk in your corpus can be embedded offline ahead of time and stored in a vector index. When a user submits a query, the system encodes only the query once and compares that vector against millions of precomputed document vectors using a lightweight math operation such as cosine similarity or inner product. Compressing an entire passage into a single vector before knowing the question makes large-scale search practical, though fine-grained wording relationships between the query and passage can get lost in compression.

How Cross-Encoders Score Query-Document Pairs

A cross-encoder concatenates the query and one candidate passage into a single input sequence and feeds them together through a transformer network, which outputs a scalar relevance score for that specific pair. Rodrigo Nogueira and Kyunghyun Cho demonstrated the effectiveness of this neural reranking pattern in Passage Re-ranking with BERT (2019), an approach frequently called monoBERT in retrieval literature.

Feeding both texts into the model together lets self-attention operate across every query token and every document token at every layer. The model can spot exact entity matches, negations, qualifiers, and multi-word dependencies directly. That joint attention comes with a steep computational catch: you cannot precompute a static document vector that already accounts for an unknown future query. Every (query, candidate) pair demands its own neural forward pass at query time, restricting cross-encoders to short candidate lists.

Where Late-Interaction Models Fit

Late-interaction architectures occupy a middle ground between single-vector bi-encoders and full cross-encoders. In ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT (SIGIR 2020), Omar Khattab and Matei Zaharia keep separate token-level vectors for each document and compare query tokens against document tokens at search time. Preserving per-token representations captures richer matching signals than a single pooled vector while avoiding the cost of running a full joint transformer pass over every query-passage pair.

Architecture How Inputs Are Encoded Can Precompute Document Representations? Query-Time Scoring Typical Pipeline Role
Bi-encoder Query and passage encoded separately into single vectors Yes (one vector per chunk) Cosine similarity or inner product via ANN search First-stage dense retrieval across large collections
Late interaction (e.g., ColBERT) Query and passage encoded separately into token-level vectors Yes (multiple token vectors per chunk) Token-level interaction across stored representations High-precision retrieval or middle-tier rescoring
Cross-encoder Query and passage concatenated and processed jointly No (requires the query at inference time) Full model forward pass per (query, passage) pair Second-stage reranker on a short candidate list

The Hard Limit of Passage Reranking: First-Stage Recall

A reranker can only reorder candidates that the first-stage retriever actually found, which means any passage omitted from the initial shortlist stays invisible to the rest of the pipeline. If your vector search or keyword index retrieves 50 chunks and none of them contain the answer, even an ideal cross-encoder can only sort 50 unhelpful chunks into a slightly cleaner order.

When I debug retrieval quality in a RAG pipeline, separating recall failures from ranking failures is always the first diagnostic step. Check whether the right passage appears anywhere in the raw first-stage candidate list before shopping for a reranker. When the relevant passage is missing altogether, the bottleneck sits upstream:

  • Poor document segmentation: Splitting tables, headers, or conditions away from their context breaks retrieval before scoring starts. Review how to chunk documents for RAG to fix structural splits.
  • Mismatched representations: Domain jargon, part numbers, or acronyms often slip past generic dense embeddings unless paired with lexical search.
  • Index and filter bugs: Stale indexes, overly aggressive metadata filters, or broken access-control rules drop valid documents prior to scoring.

Reranking solves an ordering problem inside a candidate pool that already holds the answer. Upstream retrieval fixes are required whenever the candidate pool comes back empty-handed.

When Is Adding a Reranker Worth the Extra Latency and Cost?

Adding a reranker is worth the extra inference step when first-stage search reliably pulls relevant chunks into the candidate pool while leaving weaker matches near the top of the list. Because a reranker adds complexity, teams get the clearest return when downstream accuracy depends heavily on top-k precision.

Scenarios Where Reranking Pays Off

  • Noisy top results over decent recall: First-stage retrieval surfaces the right passage somewhere in the top few dozen hits, yet raw vector similarity ranks broader summary chunks ahead of the specific clause needed.
  • Tight prompt budgets: Sending fewer, cleaner chunks to the language model saves input tokens and prevents distracting tangential text from steering generation off course.
  • Citation-sensitive tasks: Customer support bots, internal policy assistants, and documentation search tools depend on placing the exact source passage in the top few slots, and users in those workflows usually tolerate a modest wait for a better answer.
  • Hybrid search fusion: Combining keyword hits from a lexical index with semantic hits from a vector index leaves you with two lists on incompatible score scales. Running a cross-encoder over the merged candidates produces one unified relevance ranking without relying solely on heuristic rank-merging weights.

Scenarios Where Reranking Adds Little Value

  • Clean first-stage top-k: If your existing retriever already puts the right passages in the top positions for your queries, an extra model pass adds cost without changing what the language model sees.
  • Strict interactive latency budgets: Real-time voice agents, typeahead autocomplete, and sub-second UI interactions rarely have room for an extra round of transformer forward passes before generation begins.
  • Tiny high-precision candidate sets: Pipelines that retrieve only 3 to 5 tightly filtered chunks from structured metadata already send nearly the entire shortlist to the prompt.
  • Domain behavior vs. lookup issues: If the model struggles with output style or specialized task formatting rather than passage selection, review RAG vs. fine-tuning to make sure you are solving the right layer of the stack.

Operational Costs and Tradeoffs of Running a Reranker

Running a reranker introduces per-candidate inference compute, serial latency on the critical path, and an additional model dependency to operate or purchase. Evaluating these operational costs upfront keeps teams from treating rerank RAG as a free accuracy toggle.

  • Extra inference calls: Cross-encoder work scales with the number of candidates per query, the token length of each passage, and total query traffic. Scoring long chunks or deep candidate lists multiplies compute consumption quickly.
  • Serial latency: Reranking sits directly between first-stage retrieval and LLM token streaming. Every candidate batch must finish scoring before the language model can read the prompt and emit its first output token.
  • Another model in production: Teams either call a commercial hosted rerank API or deploy an open-weight reranker on their own GPUs. Either choice adds versioning, timeout handling, fallback behavior when the scorer is unavailable, and monitoring.
  • Diminishing returns on deep shortlists: As an illustrative example rather than a fixed rule, rescoring 20 candidates might capture the relevant chunks already surfaced by first-stage search, whereas rescoring 1,000 candidates with a heavy cross-encoder multiplies compute and wait time for minimal gain if the useful passages were already near the top.

Some teams skip dedicated cross-encoders and ask a general-purpose language model to score or filter retrieved passages directly, though using a generative model as a passage judge is a separate pattern that typically runs slower and costs more per candidate. Plenty of straightforward RAG applications never add a second-stage reranker at all and meet their quality targets with well-tuned chunking, metadata filters, and hybrid search.

What a Reranker Is Not

A reranker is a shortlist scoring model, distinct from storage engines, embedding generators, and truth-verification layers. Keeping its boundaries clear prevents common architectural misunderstandings:

  • Neither a vector database nor an embedding model: Embedding models turn individual texts into vectors, and vector databases index those vectors for fast lookup. A reranker stores nothing and produces relevance scores for candidate pairs on the fly.
  • No guarantee of factual accuracy or authorization: A cross-encoder estimates how closely a passage matches a query. Outdated policies, inaccurate statements, or documents a user lacks permission to view will still score near the top if they match the wording of the question; access-control filtering and content governance belong outside the reranker.
  • No substitute for corpus-specific evaluation: Public model benchmarks never reflect your exact document structure, query phrasing, or latency limits. Measuring retrieval hit rates and end-to-end answer quality on your own queries remains the only reliable way to verify whether a reranker earns its keep.

FAQ

Does every RAG pipeline need a reranker?

No. Many production RAG pipelines work well with a single retrieval stage when the candidate set is small, first-stage precision is already high, or strict latency limits apply, such as in voice interfaces and autocomplete. Teams usually add a reranker only after verifying that relevant passages appear in the broader candidate pool while weaker hits still crowd the top slots.

Can a cross-encoder search an entire document collection directly?

Running a cross-encoder across a large collection at query time is generally impractical because the model must process each query-document pair together in a separate forward pass. Document scores depend on the incoming query and cannot be precomputed into a static index. Search systems therefore rely on lexical or vector indexes first and reserve cross-encoders for rescoring a small shortlist.

In hybrid search, a lexical retriever such as a BM25-style index and a dense vector retriever return separate candidate lists with incompatible raw score scales. Passing the combined deduplicated candidates into a reranker lets a single model evaluate every passage directly against the query, producing one unified relevance ordering before the top chunks go to the language model.

What is the difference between an embedding model and a reranker?

An embedding model encodes queries and documents independently into fixed vectors so a vector index can find approximate matches across millions of items. A neural reranker, typically a cross-encoder, takes one query and one candidate passage together and outputs a direct relevance score for that specific pair, trading indexability and speed for sharper ranking accuracy on a shortlist.

Should I run an open-weight reranker or call a hosted rerank API?

Both deployment options fill the same architectural slot between first-stage retrieval and generation. Hosted rerank APIs remove GPU provisioning, scaling, and model serving work in exchange for network calls and per-request vendor billing. Self-hosting an open-weight reranker keeps data inside your own infrastructure and gives direct control over model versions, while requiring you to manage inference hardware and uptime.

Sources

Written by

Chunyu Cai

Chunyu Cai is the CEO of Lotook, LLC and an entrepreneur who writes about AI models, agents, and how they change everyday work. Chunyu holds a master's degree from Imperial College London.