Optimising RAG Retrieval: Beyond Naive Top-K

Introduction and Motivation

Retrieval-augmented generation connects the parametric knowledge stored in LLM weights with non-parametric knowledge stored in external corpora. First formalised by Lewis et al. (2020)^[1], the RAG pattern has become foundational to enterprise AI deployments where hallucination risk, knowledge freshness, and citation requirements make pure-generation architectures insufficient.

The theoretical promise of RAG is straightforward: given a user query q, retrieve a set of relevant passages P = {p₁, p₂, ..., pₖ} from a corpus C, prepend them to the context window, and condition generation on both the query and the retrieved evidence. In practice, the retrieval step is consistently the weakest link. Poor retrieval — returning passages that are semantically adjacent but factually irrelevant, missing precision-critical passages due to vocabulary mismatch, or flooding the context with redundant chunks — directly degrades generation quality in ways that improved generation models cannot compensate for.

This paper addresses that gap systematically. We describe the architectural patterns that constitute a production-grade retrieval pipeline, the engineering trade-offs at each layer, and the evaluation disciplines required to measure and iterate on retrieval quality independently of generation quality.

Failure Modes of Naive Top-K Retrieval

Naive top-K retrieval encodes both the query and all corpus chunks into a shared embedding space using a bi-encoder, computes cosine similarity, and returns the K nearest neighbours. This approach has three systematic failure modes that scale poorly as corpus size and query diversity grow.

2.1 Embedding Space Collapse

General-purpose embedding models optimise for broad semantic similarity across general English text. When applied to specialised corpora — financial filings, legal case law, medical literature, or internal enterprise documentation — the embedding space collapses: semantically similar but domain-distinct passages cluster together, while domain-critical distinctions that require specialised vocabulary are compressed out.

In one financial services deployment, we observed that queries about "credit default risk" consistently retrieved passages about "credit card rewards" at higher similarity scores than passages about credit default swaps — because the general embedding model had learned strong co-occurrence of "credit" and positive sentiment terms from its pretraining corpus. This class of failure is invisible without a labelled evaluation set.

2.2 Vocabulary Mismatch

Dense retrieval encodes queries and passages into continuous vector representations, intentionally abstracting over surface form. This abstraction fails precisely when domain-specific, low-frequency terminology carries the discriminative load. Consider a legal corpus query: "estoppel by representation in property disputes." A dense-only retrieval system is likely to return passages containing common words ("property", "disputes") at the expense of passages discussing estoppel doctrine using technical legal vocabulary that appears rarely in pretraining data. BM25 and other sparse lexical methods, by contrast, reward exact term overlap and are significantly more precise on technical, low-frequency vocabulary queries.

2.3 Context Window Pollution

Even when individual retrieved chunks are plausibly relevant, naive top-K retrieval frequently returns a set of passages that are collectively redundant — multiple chunks from the same source document discussing the same fact from slightly different angles. This redundancy consumes context window tokens without providing additive evidential value, crowding out passages that would have contributed unique supporting information and increasing generation cost without accuracy benefit.

Hybrid Dense-Sparse Retrieval

The most consistently effective improvement over naive dense retrieval is hybridisation with a sparse lexical retrieval signal. The intuition is straightforward: dense retrieval captures semantic similarity; sparse retrieval captures lexical precision. These failure modes are largely orthogonal, and their combination is reliably superior to either alone.

The standard fusion mechanism is Reciprocal Rank Fusion (RRF)^[2], which combines ranked lists from multiple retrievers without requiring score calibration across systems. Given ranked lists from n retrievers, the RRF score for a document d is:

In our production deployments, hybrid retrieval with RRF consistently outperforms dense-only retrieval by 15–28% on precision@K metrics across diverse corpora, with the largest gains on technical and specialised vocabulary queries where lexical precision matters most.

def reciprocal_rank_fusion(ranked_lists, k=60):
    """Merge N ranked lists via RRF. k=60 is empirically robust."""
    scores = {}
    for ranked in ranked_lists:
        for rank, doc_id in enumerate(ranked, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1.0 / (k + rank)
    return sorted(scores, key=lambda d: -scores[d])

class HybridRetriever:
    def retrieve(self, query: str, top_k: int=10):
        # Run retrievers in parallel (use asyncio in production)
        dense_ids  = self.dense.search(query, n=50)
        sparse_ids = self.bm25.search(query,  n=50)
        fused = reciprocal_rank_fusion([dense_ids, sparse_ids])
        candidates = [self.corpus[d] for d in fused[:50]]
        return self.reranker.rerank(query, candidates)[:top_k]

Query Understanding and Rewriting

User queries in enterprise RAG systems are frequently ambiguous, underspecified, or phrased in a register that mismatches the corpus language. Query rewriting transforms the original query before retrieval using an auxiliary LLM call. Three strategies have proven reliable in production.

4.1 HyDE: Hypothetical Document Embeddings

HyDE^[3] generates a hypothetical passage that would answer the query, then embeds that hypothetical passage rather than the original query. Because the hypothetical passage is in the same linguistic register as the corpus, it tends to retrieve more precisely than embedding the raw query. HyDE adds one LLM call per query but reliably improves recall@K by 8–15% on factual corpora.

4.2 Multi-Query Expansion

A single query often has multiple valid reformulations. Multi-query expansion generates n diverse reformulations of the original query, retrieves candidates for each, and aggregates the results via RRF. This increases recall at the cost of n additional retriever calls — usually acceptable given that retrieval is cheaper than generation.

4.3 Step-Back Prompting

Some queries are too specific for direct retrieval — the exact passage may not exist, but more general passages containing the answer do. Step-back prompting^[4] generates a more abstract version of the query, retrieves for both the specific and abstract versions, and presents both candidate sets to the reranker. This is particularly effective on multi-hop reasoning queries where the answer is distributed across multiple documents.

Multi-Stage Reranking Architectures

Bi-encoder retrieval optimises for computational efficiency: encoding queries and documents independently enables pre-computation of document embeddings and sub-linear retrieval via approximate nearest neighbour search. However, this independence is also a limitation — the model cannot attend jointly over query and document during scoring. Cross-encoders process the concatenated [query, passage] pair through a transformer and produce a single relevance score. The two-stage pattern — retrieve with a fast bi-encoder, rerank the top-N with a precise cross-encoder — captures the best of both approaches.

The reranker models we find most reliable in production are cross-encoder/ms-marco-MiniLM-L-12-v2 for general corpora, BAAI/bge-reranker-v2-m3 for multilingual settings, and domain-fine-tuned rerankers where labelled relevance data is available. The latter can improve NDCG@10 by an additional 4–9 points over generic rerankers on specialised corpora.

Contextual Compression

Even after reranking, retrieved chunks frequently contain irrelevant surrounding context: a 512-token chunk retrieved because it contains one precise, relevant sentence also delivers 480 tokens of tangential content that consumes context window budget without contributing evidential value. Contextual compression extracts the minimal relevant span from each retrieved passage before passing it to the generator.

This can be implemented as: (a) a lightweight LLM prompt that asks the model to extract the relevant portion, (b) a sentence-level relevance classifier that scores and filters individual sentences, or (c) a learned extractive compression model. Approach (a) is the most reliable and easiest to tune; approach (b) is more cost-efficient at high throughput.

Semantic Caching

Production RAG systems receive substantial query traffic that is semantically near-duplicate: users asking the same question with minor variations in phrasing. Serving these queries through the full retrieval-generation pipeline wastes compute and increases latency unnecessarily. Semantic caching stores the embeddings of previously processed queries along with their results. For each incoming query, the cache retrieves the most similar historical query and returns the cached result if similarity exceeds a threshold.

In one enterprise customer support deployment with 80,000 daily queries, semantic caching achieved a 34% cache hit rate at τ = 0.94, reducing effective LLM API costs by 28% and P95 latency for cached queries from 1.8s to 95ms.

Hallucination Detection and Grounding

Even with an excellent retrieval pipeline, LLMs can generate statements that contradict or are not supported by the retrieved passages — hallucination in the RAG setting, distinct from closed-book hallucination because the evidence for fact-checking exists in the context window.

We implement two complementary grounding mechanisms. First, claim-level attribution: the generator is prompted to produce structured output with explicit citations for each factual claim. Claims without citations, or with citations that do not support the claim upon verification, are flagged for human review or automatic regeneration.

Second, NLI-based faithfulness scoring^[5]: each sentence in the generated response is evaluated against all retrieved passages using a natural language inference model. Sentences that are neither entailed nor neutral relative to any retrieved passage are classified as unsupported and trigger regeneration, redaction, or user-facing uncertainty signals.

Evaluation Framework

A disciplined evaluation framework is prerequisite to iterative improvement of any RAG pipeline. We recommend a three-layer approach that measures retrieval quality, answer quality, and business outcomes independently — so that improvements and regressions at each layer are attributable.

Empirical Results from Production Deployments

We report aggregate results across three production deployments over a 12-month period, progressing from naive top-K retrieval to the full advanced pipeline.

Implementation Recommendations

• Start with evaluation, not architecture. Build a golden query set with human-annotated answers before modifying retrieval code. Without measurement, you cannot improve systematically.
• Add hybrid retrieval first. Dense + BM25 with RRF is the highest ROI single improvement over naive dense-only retrieval. Implement this before any other optimisation.
• Fine-tune your embedding model. With 500+ labelled query-passage pairs, fine-tune a domain-specific bi-encoder. Gains are consistently 10–20% over generic embeddings on specialised corpora.
• Size the reranker to your latency budget. A MiniLM-L6 reranker over 50 candidates adds ~25ms at P95. Profile your specific hardware before committing to a model size.
• Implement semantic caching early. The ROI is high for any system with repeated or near-duplicate query patterns. It reduces both latency and cost simultaneously.
• Treat chunking as a hyperparameter. Chunk size, overlap, and boundary heuristics (sentence-level, fixed-token, semantic-paragraph) significantly affect retrieval quality. Evaluate systematically against your golden query set.

Conclusion

Naive top-K dense retrieval is the floor, not the ceiling, of RAG system performance. The pipeline described in this paper — hybrid retrieval, query rewriting, multi-stage reranking, contextual compression, semantic caching, and NLI-based grounding — achieves answer accuracy improvements of 38–72% over baseline in our production deployments, while reducing hallucination rates to under 3%.

The critical discipline is evaluation-first development: every retrieval improvement must be measured against a stable, human-labelled golden set, and retrieval quality must be tracked independently of generation quality. Without this discipline, improvements and regressions blur together and iteration slows. The techniques described here are production-ready today. The barrier to a high-quality RAG system is not availability of the techniques — it is the engineering discipline to apply and evaluate them systematically.