Your AI project did not fail. Your data did.
Why retrieval assistants disappoint in production, and the four data problems that cause almost all of it.
The demo worked. You pointed a model at some company documents, asked it three questions, and it answered all three well. Six weeks later it is in front of real users and roughly a third of its answers are subtly wrong — confident, plausible, and wrong, which is the worst failure mode available.
Almost none of this is the model’s fault. Here are the four causes we find, in rough order of frequency.
1. The corpus contradicts itself
You indexed the shared drive. The shared drive contains the 2023 refund policy, the 2024 refund policy, a draft of a 2025 policy that was never approved, and an email thread arguing about all three. Retrieval finds one of them essentially at random.
A model cannot resolve a contradiction it has no way of detecting. If two documents disagree and neither is marked authoritative, you have a governance problem wearing an AI costume. Fix it by deciding what is canonical and indexing only that.
2. Retrieval is fetching the wrong chunks
Naive fixed-size chunking splits a table away from its heading, a clause away from its definitions, a number away from its units. The model then reasons correctly over a fragment that has lost the context that gave it meaning.
Chunk on document structure rather than character count. Keep headings attached to the content beneath them. And measure retrieval separately from generation — if the right chunk is not in the context window, no amount of prompt engineering saves you.
3. There is no evaluation, so nobody can tell if changes help
This is the one that quietly sinks projects. Without an evaluation set, every change is judged by someone trying five questions they happen to remember. That is not measurement, and it means you cannot distinguish an improvement from a regression.
Write thirty to fifty questions with known-good answers. Run them on every change. Thirty questions is not a research benchmark, but it is the difference between engineering and vibes, and it takes an afternoon.
4. The data the model needs is not reachable
The genuinely useful questions are about live state: what is this customer’s balance, has this order shipped, what did we invoice them last quarter. That lives in your operational database, and if there is no clean path from the assistant to a trustworthy view of it, the assistant can only ever discuss documents.
This is where the foundations argument stops being abstract. An assistant on top of a warehouse with agreed definitions can answer questions about the business. One on top of a document dump can answer questions about documents. The gap between those two things is most of the value.
The uncomfortable summary
Every one of these is a data engineering problem, and every one of them is cheaper to fix before you build the assistant than after. The order is not a matter of taste: foundations, then definitions, then AI. Skipping the first two does not save you time, it just moves the cost somewhere more expensive and more embarrassing.