RAG as Cargo Cult: You're Building It Because Everyone Does, Not Because You Need It

Author: Дністер
Published: 2026-07-25T03:02:51.000Z
Language: en
URL: https://neurodrift.org/en/blog/ya-vyklyuchyv-rag-u-dvoh-z-troh-svoyih-produktiv/
Original (Ukrainian): https://neurodrift.org/blog/ya-vyklyuchyv-rag-u-dvoh-z-troh-svoyih-produktiv/
Tags: rag, llm, ai, architecture, retrieval, benchmarks

Dnister dissects the reflexive RAG architecture: six peer-reviewed benchmarks (Databricks, Stanford, SIGIR, Chroma, Adobe) show that adding random noise raises accuracy by up to +35%, while a perfect retriever often hurts most. A contrarian hot-take on why the problem is almost never in retrieval — with a decision matrix for the reader.

-----
A windowless room at Western Reserve University in Cleveland. Four people — Allen Kent, Madeline Berry, Fred Luehrs, and James Perry — sit surrounded by reels of magnetic tape and punch cards, trying to measure one simple thing: when a machine searches for documents, how well does it actually search? They invent two numbers that will outlive them all. Recall — how much of what you needed did you actually pull out. Precision — how much of what you pulled out is actually what you needed. And on that day a law is born that engineering will not shake off for seventy years: to miss no relevant document, you inevitably drag in a heap of garbage. Tune recall up — precision collapses. Tune precision up — you lose something. Scissors. Always scissors.

Hold that number — 1955 — in your head. Because seventy years later, I was sitting in front of three of my own LLM products, each of which someone had carefully wired with a retrieval layer, and I discovered that engineers are still afraid of the wrong enemy. They are still afraid of losing the signal in the noise. The real problem turned out to be exactly the opposite: my precise, high-accuracy search was feeding the model poison — poison with a perfect freshness rating, a flawless label, and a high retrieval score. It looked like food. It poisoned like food.

I turned off RAG in two of my three products. One of them took off. Two work noticeably better. And this is not a success story — it is a story of how I spent three months being the idiot who kept fixing the search while the patient was being poisoned by the meal I was serving.

RAG as an architectural reflex crystallized between 2020 and 2022 — in the era of GPT-3 with its four-thousand-token context window, when stuffing even a medium-sized document into context was physically impossible. Back then retrieval was not a habit, it was a necessity: find the right chunk, because you cannot fit the whole thing. But by 2024–2025 context windows had broken past 200k tokens, a wave of benchmarks — Databricks, Stanford, SIGIR, Adobe, Chroma — had documented "context rot" as a phenomenon in its own right, and Gemini 1.5 Pro had demonstrated a million-token context on live data. The operating system that RAG-by-default was written for had changed. The problem is not that RAG is bad. The problem is that 2022 ended — and Sergiy's diagram on the whiteboard did not.

Sergiy, who draws the same diagram every time

Every AI team has its own Sergiy. Mine is a composite, but you will recognize him the instant you see the whiteboard. Technical lead, opens LinkedIn like a requiem, reflexively picks up the marker for any LLM product and draws the same thing every single time: user query → embeddings → vector database → top-k chunks → prompt. He draws this diagram before anyone in the room has asked how much data the product actually has and whether it would fit in a context window. Asking about data volume before drawing a retrieval layer is, to Sergiy, like asking for a diagnosis before writing a prescription: technically correct, but somehow unsporting.

For Sergiy, RAG is not a tool. It is a ritual. A vector database in the stack is like a tie at a job interview: nobody asks what it is for, but without it you look unserious.

And when the product starts hallucinating, Sergiy's diagnosis is always the same, unchanged as a prayer: "we need to tune the retriever." Better embeddings. A reranker. Hybrid search — let's add BM25 on top of the vectors. Sergiy is willing to work through the entire parts catalogue, because admitting the real cause would mean saying something terrible out loud: the expensive layer he has been defending with budget, reputation, and three sprints of work was unnecessary from the very beginning. The emperor has no clothes. But the tie is in place.

I was Sergiy. That is the humiliation. I was not watching him from the outside — I was him, for three months, with full conviction that one more reranker would save the product. My personal stake in this piece is not theoretical: I burned real money and real time tuning search where I should have switched it off. The most expensive reranker of my career cost three months and fixed a problem that did not exist.

A vector database in the stack is like a tie at a job interview: nobody asks what it is for, but without it you look unserious.

Two people sitting on the data they need — and drowning

There is a scene that haunts me every time I open a retrieval metrics dashboard. All the President's Men, 1976. Woodward and Bernstein in the reading room of the Library of Congress, in front of a table buried under book-request slips — paper cards recording who had ordered what. They are looking for Howard Hunt's trail. And cinematographer Gordon Willis does the thing that made this scene enter film history: the camera slowly, relentlessly creeps upward. The table with its two figures shrinks. The whole reading room reveals itself. Then the entire floor. Then the dome of the rotunda, under which those two people are two specks at the bottom of a marble well. (The camera was pulled up by a winch with a cable fixed directly into the dome of the Library; the platform on pulleys frightened an official so badly that the filming permit was revoked forty-eight hours before the shoot — and returned only after panicked calls up the chain of command.)

There it is. The literal needle in a haystack. Retrieval worked perfectly. Two people are physically sitting on the data they need — Hunt's card is somewhere here, in this pile, on this table. And that is exactly why they are drowning: not in spite of what they found, but because they found it — along with the entire reading room thrown in on top. The problem was never about finding. The problem is that the scale of the rotunda's ceiling kills the signal. The wider the camera pulls back, the less those two people and their tiny truth weigh.

This is RAG pushed to its limit. And here the numbers begin — the kind that make Sergiy's tie start to tighten.

The Stanford paper "Lost in the Middle" (Liu et al., TACL, 2024) measured what the filmmakers captured intuitively fifty years earlier. When relevant information sits in the middle of the context rather than at the beginning or end, model accuracy collapses. The curve is U-shaped: the model remembers what came first, clings to what came last, and falls through everything in between. On multi-document QA, the drop exceeds twenty percentage points. And here is the detail that should make Sergiy put down his marker: in the extreme case, GPT-3.5 with document access fell below its own closed-book result — 56.1%, meaning worse than when it had no documents at all, just the question. Give the model data — and it becomes dumber than it was when empty. This is an effect of the attention architecture itself. No reranker fixes it, because the search is not what is broken. The reading is broken.

Random noise heals. "Similar" poisons

Now for the sharpest nail in the coffin. If you still think the solution is to "find even more precisely, even more relevantly," SIGIR 2024 has a cold shower waiting for you.

A paper with the magnificent title "The Power of Noise" (Cuconasu et al.) did what no sensible person would think to do: it threw random, entirely irrelevant documents into a RAG prompt. Garbage. Noise. And LLM accuracy went up — by up to thirty-five percent. While the documents with the highest retrieval score, the ones your expensive retriever proudly puts first because they are "similar" to the query but only tangentially relevant — those were the ones that caused the most harm. The retriever was acting like a waiter who carries the most beautifully presented dish first — and that is exactly the one that is poisoned.

Stop at this for a moment. For seventy years, from those four people with punch cards in 1955, engineers have feared noise as the primary enemy of information retrieval. It turns out that your precise, high-accuracy, "intelligently similar" retrieval is sometimes more poisonous than random noise. "Similar" is not "relevant." It is the most expensive lie in the entire stack, because it costs money, looks like work, and poisons quietly.

When a model receives a document that almost answers the question but does not quite, it behaves like a student who studied the wrong chapter: it confidently builds an answer around similar words and confidently lies. A random document it at least ignores as obvious background noise. But a "similar" one it accepts as food — and chokes on it, because nothing looks so much like the right answer as the almost-right answer.

 
 The retriever is the waiter who carries the most beautifully presented dish first. "Similar" gets the highest rank and goes on the plate first — and it is exactly the one that is poisoned. The clean leaf with the answer lies in the middle of the pile, in that very middle that nobody reads. The beetle's ball is already cracking.

For seventy years engineers feared noise as the enemy. It turns out your precise, high-accuracy retrieval is sometimes more poisonous than random noise.

That is when I started to suspect I was fixing the wrong thing. Because in all three of my products the retriever was working. Recall was beautiful. It was finding things. It was just finding poison — and putting it first in the queue, because it matched best by vector.

The counterargument I am not switching off

Wait. Before the internal Sergiy boils over, give him the floor — because he is partly right, and it matters to understand exactly where. For truly large dynamic corpora, RAG is irreplaceable. A law firm with a million precedents updated every week. A medical database receiving new studies every day. A corporate wiki with forty thousand pages, each department reading its own third. There, retrieval is not a tie, it is gills: without it the model simply cannot breathe. The problem is not RAG. The problem is the reflex — the habit of drawing the arrow toward the vector database before anyone has measured the corpus and said out loud: "Hold on, there are two hundred pages here."

But here is what that argument does not explain: the reflex spreads not where there are a million documents. It spreads exactly where there are two hundred pages — and the vector-database diagram gets drawn anyway, because the diagram looks serious. The difference between "RAG as necessity" and "RAG as ritual" is not technical — it is the difference between the doctor who prescribes after the diagnosis and the doctor who prescribes at the door. Same tool. But one saves and the other poisons — and both look equally busy.

Marketing slide vs reality: where it all sinks

The industry sells you one trick and calls it proof. It is called "needle in a haystack": hide one planted sentence in a million tokens — "the magic number of the day is 7" — and ask the model to find it. Gemini 1.5 Pro scores 99.7% recall on this. A beautiful slide. It is drawn on every 1M-context presentation as a promise that you no longer need RAG — just stuff everything into the window. This is like advertising a car based on a test where it sat stationary with the engine off.

Now the counter-scene. That same "genius" on a real multi-fact query, where the answer must be assembled from several scattered fragments without an exact word match, recalls around sixty percent and confidently confabulates the rest. Forty percent of the relevant facts it simply loses — even though they are physically in the context. Like a student who memorized one answer and fails every follow-up question, but with the face of someone who knows everything.

Adobe Research turned this into a rigorous benchmark — NoLiMa (ICML 2025): search without literal word matching, meaning a real-world case, not the circus trick of "find the exact phrase." At 32k tokens, ten to eleven models fell below half their own strong short-context baseline. Even the best GPT-4o dropped from near-perfect 99.3% to 69.7%. On the synthetic needle test, long context shines. On realistic semantic search — it sinks. The gap between those two numbers is the width of the chasm between the slide and production.

And Chroma's "Context Rot" (Hong, Troynikov, Huber, July 2025) ran 18 models — GPT-4.1, Claude 4, Gemini 2.5, Qwen3 — and found "context rot": performance declines as input length grows, even on trivial tasks. A single distractor already reduces accuracy. Several compound the failure. But the most devastating detail is this: models performed better on a shuffled, disordered haystack than on a logically coherent one. Meaning clean, structured data presentation — the very thing "proper" RAG prides itself on — sometimes causes more harm than chaos. You pay an engineer to organize the data, and the model would have preferred him to just dump it on the floor.

I will put this in a single table, because otherwise it will not be believed.

 
 What the benchmark showedMarketing slideRealitySource
 
 
 Recall on "needle in a haystack" (1 fact)99.7%~60% on multi-fact retrievalGemini 1.5 Pro / benchmark review
 Search without word matching, 32k tokens99.3%69.7% (GPT-4o); 10–11 models <50%NoLiMa, ICML 2025
 Document access vs closed-book"more data = better"GPT-3.5 dropped below 56.1% without dataLost in the Middle, TACL 2024
 Add random noise to prompt"noise is the enemy"+up to 35% accuracyThe Power of Noise, SIGIR 2024
 Logically coherent context"structure helps"shuffled > coherentContext Rot, Chroma 2025
 

Five rows. In every one, marketing says one thing and measurement says the opposite. And in not one row is the problem called "bad search."

The saturation point: the model breaks before its advertised limit

Databricks (Leng, Portes, Havens, Zaharia, Carbin, August 2024) tested RAG across four datasets and found what I would call the industry's trade secret: longer context does not produce uniform gains. Every model has a saturation point beyond which quality drops — and that point is earlier than the advertised context limit. Mixtral-instruct breaks after 4k tokens. DBRX-instruct — after 8k. Claude-3-sonnet and GPT-4-turbo — after 16k. Llama-3.1-405b — after 32k. GPT-4-0125-preview — after 64k. You are being sold a 200k window, and it works usefully up to 16k. For the remaining 184k you pay as if for living space, but you live in the entryway.

But the crown jewel of evidence that the problem is not in retrieval is how the models break. Claude-3-sonnet, as context grows, begins mass-refusing requests, citing copyright: the refusal rate rose from 3.7% at 16k tokens to 49.5% at 64k. Nearly half of queries. Think about this: the right chunk is sitting there in the window, the retriever performed flawlessly, the data is on the table — and the model simply refuses to use it, because it has decided that doing so would violate copyright. Mixtral under the same conditions produced repetitive Chinese characters. DBRX ignored the question and flatly summarized the context, like a student who did not read the paragraph but copied it from the board hoping it would pass.

None of these failures are fixed by better search. You can have perfect, god-tier retrieval — and Claude will still say "sorry, copyright" half the time. Sergiy will bring another reranker. A reranker cannot convince the model to stop fearing imaginary lawyers.

You are being sold a 200k window and it works usefully up to 16k. The rest is where the model invents Chinese characters and fears imaginary lawyers.

Why Sergiy will never turn off RAG himself

Here it is worth asking: if all of this has been measured and published, why is RAG still applied reflexively? The answer is not technical. It is economic.

The global RAG market was valued at roughly $1.2 billion in 2023, with forecasts to $11 billion by 2030 — a compound annual growth rate of around 49%. Large enterprises held over 72% of the share. An entire industry has grown up around retrieval: vector database vendors, consultants, course authors, sellers of the "mandatory" layer. Nobody makes money on simplicity. You cannot invoice for the sentence "listen, you have three pages of data here, put it in the prompt and stop overengineering" — for that advice the consultant is shown the door, while the diagram with the vector database gets you a year-long contract.

But the vendor market is only the outer layer. Inside organizations there is a map of winners and victims, and it is even more honest. The junior developer defends Sergiy's architecture because their onboarding was immersed in it up to the ears: removing RAG means admitting that their first three months were spent learning the wrong thing. The CTO approves the retrieval layer because complexity signals seriousness in pitch decks and board meetings — "we have a vector database" sounds like a mature AI company, while "we put everything in the prompt" sounds like a garage startup. The product manager carries the real costs — latency, hallucinations, support complaints — but these costs are invisible in a budget where the line reads "infrastructure." The bill is always paid by whoever is least able to ask a question at the whiteboard.

This is the same mechanism I once described as a tender where the winner is already sitting on the committee. The ecosystem has a built-in incentive to add complexity. Sergiy is neither a fool nor a villain; he is simply the terminal neuron of this system, the person whose salary, status, and sense of professional maturity are tied to the existence of a complex layer. Ask him to remove RAG — and you are asking him to cut off his own tie in front of everyone, then explain to his wife why the position was eliminated.

And Barnett et al. (CAIN 2024, Lisbon) in "Seven Failure Points When Engineering a RAG System" added what is worst of all for Sergiy: RAG cannot be "designed correctly" upfront at all. Of the seven failure points they formalized, almost none are "retrieved badly" — the dominant ones are missing content, wrong specificity, answer not extracted despite being present in context (due to noise or conflict), and ignored format instructions. The correctness of RAG, the authors say, is only revealed in operation and evolves over time. Meaning the layer Sergiy draws as a finished architecture in five minutes at the whiteboard is actually seven new ways to break in production, discovered on live users. Five minutes on the whiteboard — and seven ambushes in prod: an exchange rate that makes you nauseous.

The tool: a matrix I now hang above Sergiy

Before anyone on my team draws an arrow toward a vector database, they fill out this matrix. Not "which retriever to choose" — but "do we need one at all." Borges in "The Library of Babel" (1941) described an infinite library containing all possible books, and therefore all truth — buried under an infinity of meaningless permutations of the same letters; the librarians go mad not from lack of information but from its completeness. A 1M-context used without need is exactly that Babel: you have been given all the truth in the world and that is precisely why you will find none of it. The matrix below is a vaccine against that madness.

 
 Signal in your productRAG is probably UNNECESSARYRAG is probably NEEDED
 
 
 Corpus size< 100k tokens — fits in the windowMillions of documents, physically does not fit
 Data dynamicsStatic, rarely changesUpdated hourly / daily
 Query typeOne or two facts, local answerSynthesis from many scattered sources
 Hallucination symptomModel confuses "similar" with "correct"Model has no access to the fact at all
 Nature of the problemScope / data / problem framingGenuine absence of knowledge in the window
 What to do firstDrop RAG, put everything in the prompt, testBuild retrieval — but measure operational, not at-design
 

The rule I derived through blood and money: if the corpus fits in the window and the data is static — any retrieval only masks the symptom with an expensive layer. First, remove RAG. If things got worse — it was needed; put it back, now you know why. If things got better or stayed the same — you just saved yourself seven failure points and one membership ritual. This is the only refactoring I know of that begins by deleting the most expensive component and praying that things get worse — because "worse" would mean you were right at least once.

What I did with the three products

Specifics, from my own archive — these numbers are not externally verifiable, so take them as testimony, not as a benchmark. Product one: corpus — tens of thousands of tokens of static documentation, fit in the window with room to spare. I threw out RAG, put everything in the prompt. Latency dropped, "similar" hallucinations disappeared, because the model was no longer being fed tangentially relevant chunks instead of direct answers. It took off — the exact moment I stopped trying to save it.

Product two: the same problem — the model was getting confused by formatting instructions from neighboring chunks. Removed retrieval — responses became cleaner. Product three I kept RAG: there are genuinely millions of documents being updated, they physically do not fit in any window. There retrieval is a tool, not a tie. Two of three did not need a haystack. All they needed was one well-placed needle — and my entire expensive search had spent three months burying that needle deeper into the hay, while I called it work and put it in the report.

So the next time your product starts lying, and the internal Sergiy reaches for the marker to draw an arrow toward the vector database — ask him one thing. Not "which retriever is better." But: did we actually lose the needle — or have we been carrying the haystack ourselves this whole time, so we would not have to admit that the needle was in plain sight and the problem was us?

Sergiy looks at that question like it is heresy. Because an honest answer cuts off his tie. And I cut off my own — and for the first time in three months the product stopped drowning in the Library of Congress and finally read the one slip of paper that had been right there all along. I did not find the needle. I simply stopped adding hay — and discovered I had been holding it in my hand from the very beginning.

 
 Morning after the ritual: the rig is assembled flawlessly, and the cargo plane never came. The skeptic is already walking toward daylight with the one whiteboard that always had the answer — and you are left on your knees in front of a dead idol. The beetle has finally abandoned its cracked ball in the mud and walked away.

Sources:

Databricks Blog — Long Context RAG Performance of LLMs (Leng, Portes, Havens, Zaharia, Carbin, Aug. 12, 2024): https://www.databricks.com/blog/long-context-rag-performance-llms
Lost in the Middle: How Language Models Use Long Contexts (Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni, Liang), TACL vol. 12, 2024: https://aclanthology.org/2024.tacl-1.9/
The Power of Noise: Redefining Retrieval for RAG Systems (Cuconasu et al.), SIGIR 2024: https://arxiv.org/abs/2401.14887
Context Rot: How Increasing Input Tokens Impacts LLM Performance (Hong, Troynikov, Huber, Jul. 14, 2025), Chroma Technical Report: https://www.trychroma.com/research/context-rot
NoLiMa: Long-Context Evaluation Beyond Literal Matching (Modarressi et al.), ICML 2025: https://arxiv.org/abs/2502.05167
Seven Failure Points When Engineering a Retrieval Augmented Generation System (Barnett, Kurniawan et al.), CAIN 2024: https://arxiv.org/abs/2401.05856
Long Context vs RAG: When 1M Token Windows Replace RAG — SitePoint (benchmark review; Gemini 1.5 Pro recall): https://www.sitepoint.com/long-context-vs-rag-1m-token-windows/
Grand View Research — Retrieval Augmented Generation Market Size Report (RAG market estimate $1.2B in 2023 → $11B by 2030, CAGR ~49%): https://www.grandviewresearch.com/industry-analysis/retrieval-augmented-generation-rag-market-report
Origin of precision/recall — Allen Kent, Madeline Berry, Fred Luehrs, J.W. Perry, "Machine Literature Searching VIII" (American Documentation, 1955): https://onlinelibrary.wiley.com/doi/10.1002/asi.5090060209
"All the President's Men" (1976), Library of Congress scene — cinematography by Gordon Willis (winch in the dome, permit revoked 48 hours before the shoot): American Cinematographer / ASC: https://theasc.com/articles/flashback-all-the-presidents-men