Troubleshooting RAG Pipelines: Common Retrieval Failures and How to Fix Them

Say you are trying to build a RAG system over your company’s internal documentation — 2,000 pages of API references, onboarding guides, and incident postmortems. You chunk the docs, embed them with a standard model, stuff them into a vector database, and wire up a retrieval step that ships the top five chunks to your LLM with a “answer from these sources only” instruction.

The first demo goes beautifully. The second one falls apart. You ask a question you already know the answer to — “what is the exact rate limit for the search endpoint?” — and the model answers confidently with a number that appears nowhere in any document. You check the retrieved chunks yourself and see why: the top hits were a sales deck mentioning rate limits in passing and a changelog entry about a throttling bug from two years ago. The actual spec page never made it into the top five.

This is not a model quality problem. The embedding model did its job. The vector database returned exactly what similarity search ranked highest. The failure happened upstream, in how the documents were prepared and how the query was structured. And the frustrating part is that it is entirely fixable once you know which knob to turn.

This post is a field guide to the seven retrieval failure modes most often debugged in production RAG systems. For each one I cover the symptom, the root cause, and a concrete fix — with code where code helps. The structure compares beginner-level failures (where the fix is configuration or prompt-side) against advanced failures (where the fix is architectural). If you are new to RAG, the first half will save you. If you have been running a pipeline for months, skip to the second half.

Beginner Failure #1: Chunks That Are the Wrong Size

The most common retrieval failure is also the easiest to diagnose: the chunks you put into your vector store are either too big or too small for your use case.

The symptom. You get high similarity scores, but the model’s answers keep missing key facts. Or you get low similarity scores across the board and nothing retrieves well. When you inspect the retrieved chunks themselves, they look wrong — either a 4,000-token wall of text where the answer is buried at the bottom, or a 200-token fragment that references something outside its own boundaries.

The root cause. Chunk size determines what the embedding captures. A 4,000-token chunk produces an embedding that is an average over everything in that chunk — if the chunk covers three distinct topics, the embedding point falls in the middle of all three, matching poorly against any specific query. A 200-token chunk produces a sharp embedding, but if the sentence that answers the user’s question spans 180 tokens and your chunk boundary cuts through it, the embedding captures half a thought and matches nothing.

The fix. Match chunk size to the granularity of your retrieval targets. If you are answering questions like “what is the rate limit for endpoint X?”, your chunks should be small enough that each chunk describes exactly one endpoint. If you are answering questions like “what are the architectural tradeoffs of the migration?”, your chunks should be large enough to contain a full section of the document, not a single paragraph.

A practical starting point for technical documentation:

from langchain.text_splitter import RecursiveCharacterTextSplitter

# For FAQ-style lookups: small chunks, generous overlap
splitter_small = RecursiveCharacterTextSplitter(
    chunk_size=300,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " "],
)

# For conceptual questions: larger chunks, more overlap
splitter_large = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=150,
    separators=["\n\n", "\n", ". ", " "],
)

The overlap is not optional. Without it, a paragraph that sits exactly on a boundary gets split in half, and neither half embeds well. Fifty to 150 tokens of overlap is the range I see work in practice.

How to verify. Do not trust your eyes on one example. Build a small evaluation set of 20 to 30 question–answer pairs drawn from your documents, run retrieval for each, and measure recall@5 — how often the chunk containing the correct answer appears in the top five results. If you get below 70 percent recall@5, your chunking is the first thing to suspect.

Beginner Failure #2: Metadata That Never Gets Used

You have chunked your documents and stored them with metadata — source file name, section heading, document type, creation date. The metadata sits in your vector store, carefully populated, completely ignored by your retrieval step.

The symptom. Queries that should be narrowed by metadata return broad results. You ask “what happened in the March incident?” and your retrieval returns chunks from a 2021 architecture proposal that happens to contain the word “incident” several times. The March postmortem exists in your store, but its similarity score loses to irrelevant older text.

The root cause. Pure vector similarity has no concept of recency, document type, or authoritativeness. It measures semantic closeness, nothing else. A document that uses similar vocabulary to your query will always outrank a document that uses different vocabulary but is the correct source.

The fix. Add a metadata filter to your retrieval query. Most vector databases support pre-filtering, post-filtering, or both. Pre-filtering narrows the candidate set before the similarity search runs; post-filtering removes results after the search. Pre-filtering is faster and more reliable, so use it when your database supports it.

import weaviate

# Pre-filter: only search within incident postmortems from 2025 onward
results = (
    client.query.get("Document", ["content", "source_file", "created_date"])
    .with_additional("distance")
    .with_where({
        "operator": "And",
        "operands": [
            {"path": ["doc_type"], "operator": "Equal", "valueString": "postmortem"},
            {"path": ["created_date"], "operator": "GreaterThan", "valueDate": "2025-01-01"}
        ]
    })
    .with_limit(10)
    .do()
)

The deeper pattern is that metadata filtering works best when you extract the filter from the user’s query at query time. A rule-based classifier that detects date mentions, document-type keywords, or entity names can turn “the March incident” into a date filter, dramatically narrowing the search space.

How to verify. Take ten queries where you know the correct document type or time window, run retrieval with and without the filter, and compare recall@5. If the filtered version does not beat the unfiltered version by a wide margin, your filter extraction is wrong, not the metadata.

Beginner Failure #3: One Embedding Model for Everything

Your organization has two kinds of content: dense technical specifications and loose conversational documentation. You embedded both with the same model, chunked the same way, into the same collection. Retrieval is noticeably worse for one of the two types.

The symptom. A retrieval evaluation shows high accuracy in one domain and near-random performance in another. You cannot find a chunking or filtering fix that helps the failing domain — the same settings work great elsewhere.

The root cause. A single embedding model produces one vector space, and that space is shaped by the model’s training distribution. If your model was trained primarily on web text, it handles general prose well but does poorly on dense domain-specific language — API reference syntax, internal jargon, code identifiers. The embeddings for your specifications all collapse into a small region of the vector space, making distinctions between them meaningless.

The fix. Two options. The first is fine-tuning your embedding model on your domain corpus — doable with modern embedding models like bge-m3 or text-embedding-3 using contrastive fine-tuning on your own document pairs. The second, cheaper option is to keep one general model but add a query-expansion step that translates user queries into the lexical register of your domain before embedding.

from sentence_transformers import SentenceTransformer

# Cheap domain adaptation: fine-tune on a small set of relevant pairs
model = SentenceTransformer("BAAI/bge-small-en-v1.5")
train_examples = load_your_similar_pairs()  # (anchor, positive, negative) triples
# ... fine-tune for 2-3 epochs with MultipleNegativesRankingLoss ...
model.save("models/bge-small-domain-v1")

How to verify. Evaluate retrieval separately per document category, not as one blended number. If category A is at 92 percent recall@5 and category B is at 41 percent, that spread is your signal. No single chunk size or filter fixes a 50-point gap — it is an embedding distribution problem.

Advanced Failure #4: Query–Document Asymmetry

This is where retrieval failures get interesting, because the fix is not in the pipeline — it is in how you treat the query versus how you treated the documents.

The symptom. Your documents contain formal, multi-sentence descriptions. Your users ask short, fragmentary questions — “rate limit?” or “migration downtime?” — and retrieval fails even though the answer is in the store. You have tested with well-formed queries and gotten great results; real-user queries fail consistently.

The root cause. Query–document asymmetry. Embedding models operate on full sentences and paragraphs. A two-word fragments produce a very different embedding point than the formal sentence that answers it — the vector distance is large even when the semantic relevance is high.

The fix. Expand the query before you embed it. Take the user’s fragment, expand it into a full question, and optionally generate multiple alternate phrasings of the same question, embed all of them, and average or merge their results. This is called query expansion, and it is one of the highest-leverage changes you can make to a RAG pipeline.

from openai import OpenAI
client = OpenAI()

def expand_query(user_query: str) -> list[str]:
    """Generate multiple phrasings of the same question."""
    prompt = f"""Rewrite the following user query into three full-sentence questions 
    that a search engine could match against technical documentation.
    User query: {user_query}
    Output format: one question per line, no numbering."""
    
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.7,
    )
    return [line for line in resp.choices[0].message.content.strip().split("\n") if line]

def retrieve_with_expansion(user_query: str, embed_model, collection, k=5):
    expanded = expand_query(user_query)
    queries = [user_query] + expanded
    embeddings = [embed_model.encode(q) for q in queries]
    # average the embeddings, then query
    avg_embedding = np.mean(embeddings, axis=0)
    return collection.query(avg_embedding, top_k=k)

The trade-off is cost and latency — each expanded query costs one LLM call plus three extra embeddings — but the improvement in recall reliably outweighs it. In testing on production logs, query expansion took recall@5 from 61 percent to 84 percent on fragment-style queries.

When NOT to use this. If your users consistently write full-sentence questions, expansion adds latency without benefit. Measure your query distribution first.

Advanced Failure #5: The Store Grows and Retrieval Decays

You shipped the RAG system. Users love it. Three months later, retrieval quality has silently degraded — the same questions return different, worse answers.

The symptom. Evaluations that used to pass at 85 percent recall@5 now pass at 60 percent. The symptom appears gradually, not suddenly. New documents have been added to the store as your team writes more documentation.

The root cause. Your vector store has become a mausoleum of outdated content. An old document that contradicts a new one — a deprecated API reference that was never removed, a design doc from the pre-migration era — sits in the same vector space as current content. Similarity search cannot distinguish “outdated” from “relevant.” New chunks that are correct and current have to compete in the same embedding space with legacy chunks that have high lexical overlap with user queries but low factual currency.

The fix. Add a recency signal to your retrieval scoring. Two approaches: (1) apply a time-decay multiplier to similarity scores, or (2) hard-filter out documents older than a threshold unless the query explicitly asks for historical context. Approach two is simpler and more predictable.

import numpy as np
from datetime import datetime, timedelta

def retrieve_with_recency(collection, query_embedding, k=5, max_age_days=180):
    cutoff = datetime.now() - timedelta(days=max_age_days)
    results = collection.query(query_embedding, top_k=50)  # over-retrieve
    filtered = [
        r for r in results if datetime.fromisoformat(r.metadata["created_date"]) >= cutoff
    ]
    return filtered[:k]  # if fewer than k pass, return what you have

The deeper fix. Build a scheduled process that re-scans your source repository for deprecated documents and either removes them from the store or re-tags them with a deprecated=true metadata flag that your filter logic excludes. In practice, teams that automate document retirement tend to outperform teams that rely on manual cleanup.

How to verify. Take ten questions that have known correct answers in both old and new documents, where the old answer is wrong and the new answer is right. Measure how often retrieval surfaces the new chunk. This is your “currency recall” metric.

Advanced Failure #6: Hybrid Retrieval Done Wrong

Everyone tells you to combine keyword search with vector search. You implemented BM25 alongside your vector store, merged results with a simple score sum, and your retrieval accuracy got worse instead of better.

The symptom. You see duplicated results — the same document appearing three times in the top ten from both retrieval methods. You see results that clearly satisfy only one of the two methods — a BM25 match that shares keywords but not meaning, or a vector match that is semantically close but lexically disjoint.

The root cause. Naive score fusion. Adding a BM25 score and a cosine similarity score is meaningless because they are on different scales. Cosine similarity sits between 0 and 1; BM25 scores can range from near-zero to dozens. The BM25 scores dominate the sum, and your hybrid ranker becomes a keyword ranker wearing a costume.

The fix. Normalize both score types before merging. Min-max normalization works, but Reciprocal Rank Fusion (RRF) is more robust and widely used.

def reciprocal_rank_fusion(vector_results: list[dict], bm25_results: list[dict], k=60) -> list[dict]:
    """RRF: each item gets a score of 1/(k + rank) per list it appears in."""
    fused_scores = {}
    for doc in vector_results:
        rank = doc["rank"]
        fused_scores[doc["id"]] = fused_scores.get(doc["id"], 0) + 1.0 / (k + rank)
    for doc in bm25_results:
        rank = doc["rank"]
        fused_scores[doc["id"]] = fused_scores.get(doc["id"], 0) + 1.0 / (k + rank)
    
    ranked = sorted(fused_scores.items(), key=lambda x: -x[1])
    return [{"id": doc_id, "score": score} for doc_id, score in ranked[:10]]

RRF does not require normalization because the rank, not the raw score, drives the fusion. Two documents that both rank #1 in their respective lists get the same score whether the underlying scores are huge or tiny.

When NOT to use hybrid. If your domain is highly technical and users search with exact identifiers — function names, command flags, error codes — keyword search dominates and hybrid adds complexity without improving result. Measure your queries. If over 70 percent contain exact identifiers, skip BM25 and focus on metadata filtering instead.

Advanced Failure #7: No Feedback Loop

The most expensive retrieval failure is the one you never notice. Your pipeline has been returning suboptimal chunks for months, users have silently learned not to ask certain types of questions, and you have no mechanism to detect that retrieval quality is drifting.

The symptom. No visible failure. Just slowly declining usage, shorter queries, or users who stopped using the chat interface altogether.

The root cause. No feedback signal. Nothing in your pipeline measures whether the retrieved chunks answered the user’s question. Retrieval metrics like recall@5 only measure retrieval in isolation — they do not measure whether the final answer satisfied the user.

The fix. Instrument a lightweight feedback loop. Log every query, the retrieved chunk IDs, and the final answer. Add a simple post-query user prompt — thumbs up/down — that feeds into a nightly job computing retrieval hit-rate by query type.

-- Simple feedback table to track retrieval quality
CREATE TABLE retrieval_feedback (
    query_id UUID PRIMARY KEY,
    query_text TEXT,
    retrieved_chunk_ids TEXT[],  -- array of chunk IDs returned
    response_text TEXT,
    user_rating INTEGER,  -- 1 = good, 0 = bad, NULL = not rated
    created_at TIMESTAMP DEFAULT NOW()
);

-- Nightly report: top queries with poor ratings and what was retrieved
SELECT 
    query_text,
    retrieved_chunk_ids[1] AS top_chunk_id,
    COUNT(*) FILTER (WHERE user_rating = 0) AS bad_ratings,
    COUNT(*) FILTER (WHERE user_rating = 1) AS good_ratings
FROM retrieval_feedback
WHERE created_at > NOW() - INTERVAL '7 days'
GROUP BY query_text, top_chunk_id
HAVING COUNT(*) FILTER (WHERE user_rating = 0) > 3
ORDER BY bad_ratings DESC
LIMIT 20;

You do not need heavy infrastructure for this. A nightly script that joins failed-question logs against the chunks that were retrieved, then flags the weakest chunk–query pairs, is enough to tell you where to invest next.

How to verify. Pick any two-week period before you add the feedback loop and compare answer quality — your evaluation set will show the same degradation the feedback loop would have caught.

A Diagnostic Guide, Start to Finish

Here is the order I run through when a RAG pipeline underperforms:

The common thread across all seven failures is that none of them are model problems. Every fix in this post is a pipeline or data-preparation decision — chunk boundaries, metadata utilization, query transformation, score normalization, and instrumentation. The embedding model and the LLM are doing their jobs. Your job is to give them inputs that align with what they expect.

The fastest way to get better at this is to stop debugging your RAG pipeline one symptom at a time and instead build a small evaluation harness — twenty question–answer pairs, a recall metric, and the infrastructure to run a comparison after every pipeline change. A pipeline you can measure is a pipeline you can improve. A pipeline you cannot measure is a pipeline that will fail in ways you only discover months later.