What breaks in document AI

Missing documents and recall

18 of 18 quotes

• “Chat refuses with “available sources are insufficient” even when retrieval and the thought trace contain the answer”

• “If all retrieved chunks are timetables, the LLM can't answer even though the correct information exists in the knowledge base.”

• “the system might return no results or only IDs 1 and 5, falling far short of the actual ten.”

• “A large subset of matching documents are excluded from retrieval.”

• “I found that when I upload a large number of documents to the knowledge base, the AI sometimes cannot retrieve all the relevant content, so its answers may be incomplete.”

• Hit this one too? We are building for it.

• “The retrieved chunks are relevant to the query but miss necessary details spread elsewhere.”

• “The main issue was that answers were obvious in the documents but the system was just not retrieving them in a reliable way.”

• “Top-1 hit rate sat around 60% on our eval set, which sounds fine until you realize the wrong 40% was the system confidently returning similar-but-wrong documents on policy questions.”

• “"Total unpaid invoices from last quarter": a vector search returns chunks from random documents, not an answer.”

• “the questions they would like to ask are very... broad? Like "How many Xs have Y". Stuff that would require a human to review almost every document.”

• “The system retrieves semantically relevant content about the entity, but fails to prioritize results within the intended time window, even when increasing K.”

• “It pulls the completely wrong documents”

• “the performance in recall testing shows that correct results are ranked relatively low; the same issue occurs in actual usage.”

• “In several cases, Ragflow does not surface information that we know exists in the underlying documents.”

• “the amount of data synchronized in the ragflow is different from that of the original data.”

• “It is functional at RAG stuff on internal docs but definitely not good - not sure how much of this is Copilot vs corporate disarray and access controls.”

• “When we first built our RAG, it showed promising results (around 90% recall on large knowledge bases). However, when the first actual users tried it, it could barely answer anything (closer to 30%).”

• “chunks from documents 31+ do not contain `document_metadata`.”

• Hit this one too? We are building for it.