Cross-encoder reranking, how AskVault orders retrieved evidence

Written by Joseph Russo, Principal Architect, Networks and Tech, AskVault · Reviewed by Allison Chan

Last updated: June 13, 2026 · 5 min read

TL;DR

Vector search is fast but approximate, so the right chunk sometimes lands 4th or 5th instead of 1st. AskVault reranks the top candidates by true relevance before the answer is written, on every plan, by default, with nothing to configure.

Why the order of retrieved chunks matters

The first-stage retriever (vector search plus keyword search) is built for speed. It compares a precomputed representation of each chunk against the question, which is fast enough to search a knowledge base of thousands of chunks in a fraction of a second. The trade-off is precision: because each chunk was processed without knowing the question, the match is good at "this chunk is broadly on-topic" and weaker at the fine distinctions that decide a correct answer:

• Negation. "Does the Free plan include WhatsApp?" and a chunk that says "WhatsApp is not available on Free" look very similar to a first-stage retriever. Both are about Free and WhatsApp. It struggles to tell the one that confirms from the one that denies.
• Multi-condition questions. "Refund window for annual plans cancelled in the first week" needs a chunk that satisfies all three conditions. The retriever is happy to rank a chunk that strongly matches two of the three.
• The right chunk ranked too low. Several chunks are plausibly relevant, but the one that truly answers the question is ranked fourth or fifth. The model reads the top of the context first and can anchor on the wrong chunk.

In each case the retriever found roughly the right region of your knowledge base. It just did not order the candidates well enough. Reranking fixes the order.

What reranking does

Reranking re-reads the question and each candidate chunk together, then reorders the candidates by how well each one actually answers that question. Because it looks at the question and the chunk at the same time, instead of comparing two things that were processed separately and in advance, it is far better at the negation, multi-condition, and fine-ranking cases above.

It is more careful, and more costly, than first-stage retrieval, so it cannot be run over an entire knowledge base for every query. The standard pattern, and the one AskVault uses, is two stages:

1. Retrieve a candidate shortlist cheaply with the fast first-stage retriever. This narrows a knowledge base that can hold up to 100 MB of content down to a small set of likely-relevant chunks.
2. Rerank that shortlist carefully, then keep the best 5 of them for the answer.

You get the first stage's speed across everything and the rerank stage's precision exactly where it matters: on the short list that is about to become the answer. A chunk the first stage ranked low, but that actually answers the question, can be promoted into the final set instead of being dropped.

What it means for your answers

Reranking is one of the quieter reasons AskVault's Chat Playground and live customer answers hold up on hard questions. It improves three things in particular:

• More accurate on complex questions. Negation, multi-part, and "which of these is the right one" questions are exactly where first-stage retrieval slips and reranking helps most.
• Better citations. The agent is more likely to cite the page that actually contains the answer rather than an adjacent page that was merely similar.
• No reliability cost. Reranking can only reorder candidates the retriever already found. If the rerank step ever has trouble, AskVault falls back to the retrieved order and answers normally. It can never take chat down, and it never invents new content.

It runs automatically across all 13 channels AskVault supports, during the 14-day Free plan trial and on every paid plan through Enterprise: Starter, Growth, and Business included. There is nothing to switch on, tune, or pay extra for.

AskVault vs single-stage retrieval tools

Many chatbot tools stop at single-stage vector search: they return the chunks that are broadly on-topic, in whatever order the fast index produced, and rely on the language model to sort out which one actually answers the question from its position in the prompt. That works on easy questions and degrades on the hard ones. A rerank pass reorders the candidates by true relevance before the model ever sees them.

FAQ

Do I need to turn reranking on or configure it?

No. It runs by default on every workspace, with nothing to set up.

Does reranking slow down answers?

It adds only a small overhead, far less than the time the language model spends writing the answer, so it is not something your customers perceive, whether a workspace handles 10 queries/day or 10,000.

What happens if the rerank step fails?

AskVault falls back to the order the retriever produced and answers normally. Reranking only ever improves the order; a problem with it degrades to "no reranking," never to "no answer."

Is reranking the same as the initial search?

No. The initial search is a fast, approximate pass over your whole knowledge base. Reranking is a slower, more careful second look at only the shortlist that search produced, used to put the best evidence first.

Related guides

• How vector databases work
• Query decontextualization for multi-turn chat
• Chunking strategies for production RAG
• How AskVault prevents AI hallucinations
• What is Retrieval-Augmented Generation (RAG)?

See RAG-grounded answers in action

Every AskVault answer is grounded in the source documents you upload. Never hallucinated.

Try AskVault Free → Read the API docs

Was this page helpful? Yes No