Hybrid Search — Dense + Sparse + Re-Ranking — Semantic Search & Embeddings

Hybrid Search — Dense + Sparse + Re-Ranking

What you will learn

Understand why pure dense search misses exact-match queries
Implement BM25 sparse retrieval in Python
Combine dense and sparse results with Reciprocal Rank Fusion
Add a cross-encoder re-ranker as a final quality step

The problem with pure semantic search

Dense embeddings are excellent at understanding meaning but poor at exact matches. A search for "GPT-4o" will not reliably return the document that literally mentions "GPT-4o" if the embedding model has not seen that string before. Rare product names, identifiers, code snippets, and technical terms all suffer.

The solution is hybrid search — combine dense (semantic) retrieval with sparse (keyword) retrieval, then re-rank.

Hybrid Search Pipeline — three stages

  Query: "GPT-4o multimodal capabilities"
       |
       +---> Dense retrieval (embeddings)   -> Top 20 results
       |
       +---> Sparse retrieval (BM25)        -> Top 20 results
       |
       v
  Reciprocal Rank Fusion (RRF)              -> Combined Top 20
       |
       v
  Cross-encoder re-ranker                   -> Final Top 5
       |
       v
  LLM context

BM25 — sparse retrieval

BM25 scores documents by term frequency and inverse document frequency. It finds documents that contain the exact query terms, weighted by how rare those terms are across the corpus.

from rank_bm25 import BM25Okapi

corpus = [
    "MCP servers implement OAuth 2.0 for authentication",
    "Tools are callable functions; resources are readable URIs",
    "LangGraph builds agent flows as explicit directed graphs",
    "GPT-4o supports multimodal input including images and audio",
    "HNSW enables approximate nearest neighbour search",
]

# Tokenise
tokenised = [doc.lower().split() for doc in corpus]
bm25 = BM25Okapi(tokenised)

query = "GPT-4o multimodal"
scores = bm25.get_scores(query.lower().split())

ranked = sorted(enumerate(scores), key=lambda x: x[1], reverse=True)
for idx, score in ranked[:3]:
    print(f"[{score:.3f}] {corpus[idx]}")
# [1.23] "GPT-4o supports multimodal input..."  <- exact term match

Install: pip install rank-bm25

Reciprocal Rank Fusion (RRF)

RRF combines two ranked lists into one without needing to calibrate scores across different systems. It only uses the rank position (1st, 2nd, 3rd...) not the raw score values.

def reciprocal_rank_fusion(
    dense_ranking: list[str],
    sparse_ranking: list[str],
    k: int = 60,
) -> list[tuple[str, float]]:
    scores: dict[str, float] = {}

    for rank, doc_id in enumerate(dense_ranking):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)

    for rank, doc_id in enumerate(sparse_ranking):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)

    return sorted(scores.items(), key=lambda x: x[1], reverse=True)


# Example: two systems return different ranked lists for the same query
dense_ids  = ["d4", "d3", "d1", "d2", "d5"]  # embeddings ranked these
sparse_ids = ["d4", "d1", "d5", "d3", "d2"]  # BM25 ranked these

fused = reciprocal_rank_fusion(dense_ids, sparse_ids)
for doc_id, score in fused[:5]:
    print(f"{doc_id}: {score:.4f}")
# d4 appears first in both -> highest combined score

The k=60 constant prevents top-ranked results from dominating too heavily. A common default.

Cross-encoder re-ranking

A bi-encoder (like the embedding model) encodes query and document separately and compares them. A cross-encoder takes both query and document together and outputs a relevance score — much more accurate but too slow to run on thousands of candidates.

The pattern: use fast retrieval (dense + sparse) to get top-50, then use a cross-encoder on just those 50 to re-rank to the final top-5.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

candidates = [
    "MCP servers implement OAuth 2.0 for authentication",
    "GPT-4o supports multimodal input including images and audio",
    "LangGraph builds agent flows as explicit directed graphs",
]
query = "how does GPT-4o handle images?"

pairs = [(query, candidate) for candidate in candidates]
scores = reranker.predict(pairs)

ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
for text, score in ranked:
    print(f"[{score:.3f}] {text}")

Install: pip install sentence-transformers

Chapter summary

Dense search misses exact terms (rare tokens, product names, code); BM25 catches them
RRF merges two ranked lists using only position, not raw scores — no calibration needed
Cross-encoder re-ranking is expensive but highly accurate — apply only to top-N candidates
The standard pattern: dense + BM25 → RRF → cross-encoder → LLM context

Check your understanding

Give an example of a query where pure dense search would fail but BM25 would succeed.
Why does RRF use rank position instead of raw retrieval scores?
Why not run the cross-encoder on all 10,000 documents in the corpus?