Why Your RAG Retrieval Is Bad: Hybrid Search, RRF and Reranking

Last updated: August 2026

Every disappointing RAG project I get called into looks the same from outside. The demo worked. Then someone asked a real question, the assistant answered confidently from the wrong paragraph, and trust died inside a week. The reflex is to blame the model and go shopping for a bigger one.

Open the traces instead. The generator usually did fine with the three chunks it was handed, and those chunks were wrong. The failure is upstream: a single-shot vector search over documents chopped into fragments with no memory of where they came from. That is a data engineering problem with a known fix. If you are still choosing between retrieval and fine-tuning, that is a different question, covered in RAG vs Fine-Tuning Enterprise AI.

Why does single-shot vector search fail on enterprise documents?

Dense retrieval is lossy on purpose. You compress a chunk into a fixed-length vector and ask cosine similarity to preserve what mattered. Fine for topical similarity. For what enterprise users actually search on, it degrades three ways.

• Exact tokens get smeared. Part numbers, error codes, GL accounts, policy forms like HO-3, clause numbers. They carry almost no semantic signal, so the embedding barely encodes them. Ask for form HO-3, get homeowners insurance in general.
• The chunk forgot its document. A chunk reading "the deductible is $2,500 for this peril" is unretrievable and unusable. Which policy? Which peril? The splitter threw away the title, header and governing form.
• One shot, no recovery. If the query vector lands in the wrong neighbourhood there is no second opinion and no way for the system to notice it retrieved garbage.

Fix the chunk before you fix the retriever

The highest-leverage change happens at ingestion. Before embedding a chunk, generate a short context for it against the whole document and prepend that. You embed the enriched text and index it for keyword search. Anthropic published this as Contextual Retrieval, with numbers from its own evaluation:

• Contextual embeddings alone. Top-20 chunk retrieval failure rate dropped from 5.7% to 3.7%, a 35% reduction.
• Plus contextual BM25. Down to 2.9%, a 49% reduction.
• Plus a reranking pass. Down to 1.9%, a 67% reduction.
• Cost of the enrichment pass. Anthropic put the one-time cost at $1.02 per million document tokens using prompt caching, since each document is cached once and reused across all its chunks.

Those are Anthropic's results on Anthropic's eval set, not a benchmark you will reproduce. The shape transfers; the magnitude depends on how badly your chunks lost context. Also from that write-up: under roughly 200,000 tokens of corpus, skip retrieval and put it all in the prompt.

-- Step 1: split documents into chunks.
-- GOTCHA: chunk_size here is CHARACTERS, not tokens. Snowflake recommends
-- chunks of no more than 512 tokens (about 385 English words) for Cortex Search,
-- so ~1,500 characters of English prose is a sane starting point. Measure it.
CREATE OR REPLACE TABLE docs_chunked AS
SELECT
    d.doc_id,
    d.doc_title,
    d.body_text,
    c.index          AS chunk_seq,
    c.value::VARCHAR AS chunk_text
FROM docs d,
     LATERAL FLATTEN(
       input => SNOWFLAKE.CORTEX.SPLIT_TEXT_RECURSIVE_CHARACTER(
                  d.body_text, 'markdown', 1500, 250)
     ) c;

-- Step 2: generate a short, chunk-specific context against the WHOLE document,
-- then prepend it. The enriched text is what you embed AND what you index for
-- keyword search. Model availability is region dependent, so check the Cortex
-- model availability table before hardcoding a model name.
CREATE OR REPLACE TABLE docs_contextual AS
WITH ctx AS (
    SELECT
        doc_id,
        chunk_seq,
        chunk_text,
        AI_COMPLETE(
          'claude-haiku-4-5',
          '<document>' || body_text || '</document>' ||
          'Here is the chunk we want to situate within the whole document: ' ||
          '<chunk>' || chunk_text || '</chunk>' ||
          'Give a short succinct context to situate this chunk within the ' ||
          'overall document for the purposes of improving search retrieval ' ||
          'of the chunk. Answer only with the succinct context and nothing else.'
        ) AS chunk_context
    FROM docs_chunked
)
SELECT
    doc_id,
    chunk_seq,
    chunk_text,
    chunk_context,
    chunk_context || '\n\n' || chunk_text AS enriched_chunk
FROM ctx;

Two traps there. SPLIT_TEXT_RECURSIVE_CHARACTER takes chunk_size in characters while every chunking recommendation is written in tokens, so people set 512 thinking tokens and get useless fragments. And a chunk longer than the embedding model's context window is truncated before embedding, while the full text is still used for keyword retrieval. Silent truncation on the vector leg only is unpleasant to debug.

Hybrid retrieval for enterprise RAG: what BM25 adds that embeddings can't

The case for hybrid retrieval is not that BM25 beats embeddings. It is that the two fail on different queries, and the failures are weakly correlated, so the union of their candidate lists beats either alone. Candidate recall is the ceiling on everything downstream: a reranker cannot rescue a document that was never retrieved.

Run both legs, take roughly 100 candidates from each, let fusion sort it out. The lexical leg costs one index lookup: the cheapest recall you will ever buy.

How Reciprocal Rank Fusion works, and why k = 60

You cannot merge two ranked lists by adding their scores. Cosine similarity lives in [-1, 1], BM25 is unbounded and corpus-dependent, and per-query normalisation is unstable when one list is short or flat. Reciprocal Rank Fusion sidesteps all of it by keeping ranks and discarding scores.

The formula, from the original SIGIR 2009 paper by Cormack, Clarke and Buettcher: for each document d, sum 1 / (k + r(d)) over every ranked list that returned it. The paper states k = 60 was fixed during a pilot investigation and never altered, that it was near-optimal, and that the choice was not critical. The constant exists, in their words, to mitigate the impact of high rankings by outlier systems.

-- Reciprocal Rank Fusion over a lexical candidate list and a dense candidate list.
-- k = 60 is the constant from the original RRF paper (Cormack, Clarke, Buettcher, SIGIR 2009).
WITH q_vec AS (
    SELECT AI_EMBED('snowflake-arctic-embed-l-v2.0',
                    'water damage deductible under form HO-3') AS qv
),
lexical AS (
    -- NOTE: Snowflake's SEARCH() returns BOOLEAN, not a relevance score.
    -- There is no built-in BM25 ranking here, so the ordering below is a
    -- stand-in. For a real scored lexical leg, use Cortex Search or an
    -- external full-text engine and load its ranks into this CTE.
    -- SEARCH() also rejects column references for the search string: it must
    -- be a literal, which is why the query text is templated in twice rather
    -- than joined in from a CTE.
    SELECT c.chunk_id,
           ROW_NUMBER() OVER (ORDER BY c.doc_authority DESC, c.chunk_id) AS rnk
    FROM chunks c
    WHERE SEARCH(c.enriched_chunk,
                 'water damage deductible under form HO-3',
                 SEARCH_MODE => 'OR')
    QUALIFY rnk <= 100
),
dense AS (
    SELECT c.chunk_id,
           ROW_NUMBER() OVER (
             ORDER BY VECTOR_COSINE_SIMILARITY(c.embedding, v.qv) DESC
           ) AS rnk
    FROM chunks c, q_vec v
    QUALIFY rnk <= 100
),
unioned AS (
    SELECT chunk_id, rnk FROM lexical
    UNION ALL
    SELECT chunk_id, rnk FROM dense
)
SELECT
    chunk_id,
    SUM(1.0 / (60 + rnk)) AS rrf_score,
    COUNT(*)              AS lists_hit   -- 2 means both legs agreed
FROM unioned
GROUP BY chunk_id
ORDER BY rrf_score DESC
LIMIT 150;   -- this is the candidate set you hand to the reranker

The honest tradeoff: RRF discards magnitude, so a rank-1 hit at 0.95 cosine and one at 0.31 contribute identically. That is a feature when scores are uncalibrated, which is most of the time; with labelled data, weighted fusion tuned on your own judgements will edge it out. Keep the agreement count too: two independent retrievers returning the same chunk is a strong relevance prior.

Do you actually need a cross-encoder reranker?

Your retriever is a bi-encoder: query and document encoded separately, which is why document vectors can be precomputed and search scales to millions of chunks. A cross-encoder puts the query and one passage through the same forward pass, so it sees the interaction. More accurate, impossible to precompute, and linear in candidates.

So retrieve wide and cheap, rerank narrow and expensive. Anthropic retrieved 150 chunks and reranked to 20; roughly 100 to 200 in and 10 to 25 out is where most systems land. You need this when the corpus is full of near-misses, chunks that are topically identical but differ on the detail that matters, like one policy across five product lines. Skip it when recall@20 is already near 1.0.

One operational warning. Reranker model families turn over, and relevance scores are not comparable across versions. Cohere currently lists rerank-v4.0-pro and rerank-v4.0-fast alongside the older rerank-v3.5, having already retired its v2.0 rerank models on a published shutdown date. Nothing guarantees a successor puts the same passage at the same score, so every threshold has to be re-tuned on the new model. Filter by rank and you survive upgrades. Filter by score > 0.7 and one day it silently returns nothing, or everything.

On Snowflake, Cortex Search reranks by default and you can disable it per query. The docs put the saving at 100 to 300 milliseconds on average, noting that both the saving and the quality loss vary by workload.

Query rewriting is the cheapest thing in front of the stack

Users do not type well-formed retrieval queries. They type fragments, pronouns and follow-ups. In a chat interface, turn four is "what about for commercial?", meaningless standalone, and that is what most pipelines send to the retriever.

• Conversational rewrite. One small-model call turning the conversation so far plus the latest message into a standalone query. Fixes more multi-turn bugs than any amount of embedding-model shopping.
• Deterministic term expansion. A curated glossary expanding internal acronyms and codenames before retrieval. No model call, no latency, no non-determinism. Every enterprise has fifty of these and none are in the model's training data.
• Multi-query generation. Ask for three phrasings, retrieve for each, fuse with RRF. Good where users ask vaguely and recall is the bottleneck.

The trap is in the third: fan-out multiplies everything downstream. Rerank three lists separately and you just tripled your reranker bill. Fuse into one list with RRF, dedupe, rerank once.

Building it on Snowflake: Cortex Search or a DIY VECTOR pipeline?

If the documents already live in Snowflake, Cortex Search gives you most of this managed: vector search, keyword search and semantic reranking, with the index kept fresh against a TARGET_LAG.

-- Managed hybrid retrieval + semantic reranking, built on the enriched chunks.
CREATE OR REPLACE CORTEX SEARCH SERVICE policy_docs_search
  ON enriched_chunk
  ATTRIBUTES doc_id, product_line, effective_date
  WAREHOUSE = search_build_wh
  TARGET_LAG = '1 hour'
  EMBEDDING_MODEL = 'snowflake-arctic-embed-l-v2.0'
  AS (
    SELECT
      enriched_chunk,
      chunk_id,
      doc_id,
      product_line,
      effective_date
    FROM docs_contextual_indexed
  );

-- Tune the fusion weights and inspect results. SEARCH_PREVIEW is for testing
-- and validation only; serve real traffic from the Python or REST API.
SELECT PARSE_JSON(
  SNOWFLAKE.CORTEX.SEARCH_PREVIEW(
    'policy_docs_search',
    '{
       "query": "water damage deductible under form HO-3",
       "columns": ["enriched_chunk", "doc_id"],
       "filter": {"@eq": {"product_line": "homeowners"}},
       "limit": 20,
       "scoring_config": {
         "weights": {"texts": 3, "vectors": 1, "reranker": 2}
       }
     }'
  )
)['results'] AS results;

The weights block is your fusion control: relative weights for the keyword leg, the vector leg and the reranker, each defaulting to 1.0. Turn texts up for corpora full of identifiers; "reranker": "none" switches reranking off. The trap is SEARCH_PREVIEW itself. Snowflake documents it as being for testing and validation, explicitly not intended for serving queries in an end-user application, with higher latency and a response size limit. It is also the only SQL-callable way in, so it ends up in internal apps constantly. Serve real traffic from the Python or REST API.

Going DIY with VECTOR columns, three documented constraints shape the design. VECTOR(INT|FLOAT, n) caps n at 4,096 dimensions. VECTOR_COSINE_SIMILARITY is optimised in a way that reduces floating point precision and carries a documented margin of error up to 1e-4, so never build exact-equality logic or tie-breaks on raw similarity. And SEARCH() returns a BOOLEAN, not a score, so there is no built-in BM25 ranking to fuse with: matching, not ranking. It also takes a literal search string only, not a column reference, so the query text has to be templated in by the caller.

Match the column dimension to the model or the insert fails. snowflake-arctic-embed-l-v2.0 outputs 1024 dimensions, snowflake-arctic-embed-m-v1.5 outputs 768. Changing model later means a full re-embed, so decide with an eval, not a vibe.

What each stage costs per answer

Every technique buys accuracy with latency, tokens or both. Treat the latency column as shape, not benchmark, apart from the documented figure.

Everything above the last row adds a bounded amount. The agentic loop multiplies, which is the difference between a sub-second answer and a twelve second one.

Agentic RAG: when does the retrieve-judge-retrieve loop pay for itself?

The layer above hybrid retrieval lets the model drive: decompose the question, retrieve per sub-query, judge whether what came back is sufficient, decide whether to go again. More capable, more expensive: each hop re-reads accumulated context, so tokens grow faster than hops.

It earns its cost on questions no single chunk can answer. "How did our retention policy change between the 2023 and 2025 handbooks" needs two passages from two documents, compared. It does not on single-document lookup, which is most enterprise traffic. Build a router: easy 80% down the single-pass path, loop for the rest.

• Hard-cap the hops at two or three, not "until the model is satisfied". A model that cannot find the answer keeps looking.
• Budget tokens per answer and fail closed with a partial answer plus citations rather than quietly spending ten times your estimate.
• Dedupe chunk IDs across hops. Otherwise hop two retrieves most of what hop one had and you pay twice for the same context.
• Log the decomposition: sub-queries, chunk IDs, hop count, every answer. When a bad answer surfaces three weeks later, that log tells you whether retrieval or generation failed.

Prove it works, then watch these cost traps

End-to-end answer quality cannot separate a retriever failure from a generator failure, so measure retrieval on its own. Take 100 to 200 real questions from user logs, label the chunk IDs that should come back, then track recall@k and mean reciprocal rank on every config change.

# Retrieval metrics, measured separately from answer quality.
# Run over the same labelled question set for every config change.

# Fraction of questions whose gold chunk appears in the top k.
def recall_at_k(runs, k):
    hits = sum(1 for r in runs if set(r["gold_chunk_ids"]) & set(r["retrieved"][:k]))
    return hits / len(runs)

# Mean reciprocal rank of the first gold chunk.
def mrr(runs):
    total = 0.0
    for r in runs:
        gold = set(r["gold_chunk_ids"])
        for i, cid in enumerate(r["retrieved"], start=1):
            if cid in gold:
                total += 1.0 / i
                break
    return total / len(runs)

for name, runs in variants.items():   # dense_only, plus_lexical_rrf, plus_context, plus_rerank
    print(f"{name:24s} recall@5={recall_at_k(runs,5):.3f} "
          f"recall@20={recall_at_k(runs,20):.3f} mrr={mrr(runs):.3f}")

Usually one variant does most of the work. Ship what moves your numbers, skip what only moved someone else's. Then watch for these:

• Managed search indexes bill while idle. Snowflake's docs are explicit that Cortex Search serving cost is incurred while the service is available to respond to queries, even if none are served, charged per GB per month of indexed data including the computed embeddings.
• Changing chunk strategy re-embeds everything. Embedding cost is incremental, incurred only for added or changed documents, which sounds reassuring until you notice that re-chunking changes every document.
• Reranking recall you already have. If recall@20 is 0.98 without a reranker, pushing 150 candidates through a cross-encoder buys almost nothing and costs a round trip per query.
• Score thresholds that break silently. Model upgrades change score distributions and similarity functions carry documented precision margins. Threshold on rank, never on an absolute score.
• Preview functions in production. SEARCH_PREVIEW works, which is exactly why it ends up serving queries it was never meant to.

None of this is exotic. It is chunking, indexing, fusion arithmetic, a candidate budget and an eval set: a data engineering job with a language model at the end. Teams that treat it that way fix retrieval in weeks. Teams that treat it as model selection are still swapping models six months on.