企業知識庫接入 AI，為何答案仍不可靠？從檢索命中到可核查回答

1. Retrieval, support and correctness are different

Retrieval can find a relevant passage while the model adds an unsupported condition; faithful repetition can also reproduce an expired policy. ACL 2024’s RAGTruth annotates statements unsupported by, or contradictory to, supplied retrieval content.[1] Using RAG therefore does not establish hallucination-free answering.

Record whether required evidence was retrieved, key claims are supported, necessary parts are complete, and sources meet date and access requirements. One overall score can compare versions without identifying the repair. Different failures need different owners.

2. Retrieval improvements matter without proving answer accuracy

Anthropic’s 2024 engineering tests reported top-20-chunk retrieval failure falling from 5.7% at baseline to 2.9% with contextual embeddings plus BM25, and 1.9% with reranking added.[2] The final change is 3.8 percentage points, or about 67% relative reduction, not a 67-point improvement in answer accuracy.

These experimental results need retesting on local documents, chunking and questions. TACL 2024’s Lost in the Middle also found position-sensitive performance in the models tested.[3] It motivates evaluation of context use; it is not a fixed prediction for every 2026 long-context model.

3. A 2026 enterprise benchmark models messiness with limits

The May 2026 EnterpriseRAG-Bench preprint contains about 512,000 synthetic documents across nine enterprise-source types and 500 questions, including duplicates, misfiling, conflicts and missing information.[4] Enterprise-style content is not real customer production data; the paper discusses generation, representation and incomplete-gold-label limitations.

Basic and semantic lookup account for 300 of the 500 questions, with additional categories for multi-document reasoning, constraints, conflict, completeness and absent answers. Ask whether the test covers locally important question families as well as its overall score. Finding every active exception requires more than one-document retrieval.

4. Product architecture must connect permissions and versions

Microsoft’s connector documentation distinguishes synced content indexed into Microsoft Graph from federated connectors fetching content through MCP at query time without syncing it into Graph.[5] This is a documented architecture distinction, not an accuracy result from a customer deployment.

Both require source identity, freshness and access checks; live fetching cannot guarantee that upstream policy is valid. For an expense policy, effective dates, supersession, geography and approval authority can matter more than similarity. Assign ownership of authoritative versions and propagation of deletions or access changes rather than treating every document equally.

5. Citations support inspection, not automatic endorsement

A real citation may support only part of a paragraph. Let users locate the supporting passage and inspect dates, versions and conflicts; separate known facts, inference and unresolved points. Microsoft documents source inspection through answer citations,[5] but support for each claim still needs evaluation.

An abstention can explain missing evidence, conflicting sources and the responsible owner. Do not set thresholds solely from model self-confidence or improve accuracy by silently deleting difficult questions. Keep coverage, error rates and human workload visible against the same incoming-task denominator.

6. An original model: coverage, review and undetected errors

For N requests, answer coverage a, error fraction among answered requests e, random review fraction h and detection probability d on reviewed errors, expected answers are Na, errors Nae and undetected errors Nae(1−hd). Review is assumed representative and detected errors corrected or withheld. This is scenario arithmetic, not measured product performance.

Defaults N=1,000, a=80%, e=8%, h=25% and d=80% give 800 answers, 64 expected errors and 51.2 undetected by review. At two minutes per reviewed answer and six per unanswered request, human effort totals 1,600 minutes, about 26.7 hours. Fractional counts are expectations, not fractions of actual incidents.

Increasing review alone to 75% lowers expected undetected errors to 25.6 but raises human effort to 40 hours. With an illustrative undetected-error budget of ten, the default review conditions require an answered-error rate no higher than about 1.56%. This is not a risk-tolerance recommendation. Consequences, non-random review, false positives and repeated handling are excluded.

7. Organise local acceptance by questions and failure types

Version a question set with business-reviewed answers, required evidence, identities and dates. Include routine lookup, same-name files, old/new versions, cross-team exceptions, absence, insufficient access and exhaustive lists. Do not build the test only from conversations the system already answered correctly.

Fix data snapshots and identities, retaining retrieved passages, answers and claim-level judgements. Audit automated graders with human samples, especially costly rare categories. Record reasons for changing questions or gold answers. A synthetic-benchmark lead supports technical candidacy rather than replacing local acceptance.

8. A useful knowledge assistant exposes its boundaries

Good initial scopes have authoritative sources, controlled versions, clear questions and an owner for escalation. Narrow scope and establish confirmation for rapidly changing, cross-permission or consequential content. Model, index and access changes can alter results and should trigger relevant rechecks.

Enterprise knowledge AI creates value when users reach usable evidence faster, understand what remains uncertain and reach the right owner. Deliver retrieval, claim support, freshness and accountability together to approach a dependable knowledge service.

1. 检索命中、回答有据和事实正确，是三件事

检索可以找对一段文档，但模型仍可能多写一个未获支持的条件；模型也可能忠实复述一份已经失效的制度。2024 年 ACL 的 RAGTruth 研究专门标注了检索增强回答中不被所给内容支持或与之矛盾的表述。[1] 因此，“用了 RAG”与“没有幻觉”之间没有等号。

评估应分别记录：应找资料是否找到、关键主张是否有依据、是否回答了全部必要部分，以及来源是否在适用日期和权限范围内。一个总分有利于比较版本，却不够指导修复。不同失败需要不同负责人。

2. 检索改进有价值，但指标不能被偷换

Anthropic 在 2024 年的工程实验中，报告前 20 个片段的检索失败率从基线 5.7%，降到上下文嵌入与 BM25 组合的 2.9%；再加入重排为 1.9%。[2] 从 5.7% 到 1.9% 是下降 3.8 个百分点、约 67% 的相对下降，不是回答准确率提高 67 个百分点。

这些是特定实验的数据，仍需在自己的文档、切分方式和问题上复测。2024 年 TACL 的 Lost in the Middle 还显示，当相关信息在长上下文中的位置改变，所测模型的表现会改变。[3] 它是基础方法提醒，不是对所有 2026 年长上下文模型的固定性能预测。

3. 2026 年企业基准，更贴近杂乱，也仍有边界

EnterpriseRAG-Bench 于 2026 年 5 月发布预印本，包含约 51.2 万份合成文档、九类企业来源和 500 道问题，刻意加入近重复、错误归档、冲突和缺失信息情景。[4] 它的企业内部风格不等于真实客户生产数据，论文也明确讨论合成方式、结构和标准答案不完整的限制。

500 题中，基础查找与语义查找共 300 题，另有跨文档、约束、冲突、完整性和“没有答案”等类别。这个构成说明，采购时不仅应问总分，还应问本组织真正重要的问题族是否被覆盖。若关键任务是找到所有生效例外，单份资料命中率就不够。

4. 产品架构案例：连接内容，也要连接权限与版本

Microsoft 的官方连接器文档区分两种路径：同步连接器把外部内容索引进 Microsoft Graph；联合连接器在查询时通过 MCP 获取内容，不把内容同步进 Graph。[5] 这是可核查的产品架构差异，不是某家公司采用后的准确率证明。

两种路径都需要核查来源标识、更新时间与访问控制；实时获取也不能保证上游制度本身有效。对一份报销政策，生效日、替代关系、适用地区和审批主体可能比相似度更重要。应明确由谁确认权威版本，删除与权限变更多久传播，而不是把所有内容视作同等级知识。

5. 引用是核查入口，证据不足时应保留退出路径

一个引用链接可能真实存在，却只支持段落中的一半主张。回答界面应让人定位到具体支持片段，并看见日期、版本和冲突；对于无法确认的部分，明确区分已知、推断和待核查。Microsoft 官方资料描述了从回答引用查看源内容的机制，[5] 但具体主张是否被支持仍需测试。

拒答不应只是“我不知道”，还可说明缺少什么、哪些来源冲突、应向谁确认。不能仅凭模型自报的信心确定阈值，也不能让删除困难问题成为提高准确率的捷径。覆盖率与错误率必须使用同一任务分母，并保留转人工工作量。

6. 原创模型：复核比例、回答覆盖率与漏检错误

设每期 N 个请求，回答覆盖率 a，已回答中错误比例 e，随机抽查比例 h，抽查发现错误的概率 d。预期已回答数为 Na，错误数为 Nae，未被发现的错误数为 Nae(1−hd)。这里假设抽查具有代表性，且被发现的错误会被纠正或拦下；这是情景算术，不是产品实测。

默认 N=1,000、a=80%、e=8%、h=25%、d=80%：回答 800 个、错误期望 64 个，其中约 51.2 个未被复核发现。若复核每个回答需 2 分钟、未回答请求转人工需 6 分钟，总人工需求为 1,600 分钟，约 26.7 小时。分数计数是期望值，不是声称发生了半个真实事件。

仅将复核比例提高到 75%，漏检期望降为 25.6，但人工需求升到 40 小时。若自设漏检预算为 10，默认复核条件下允许的回答错误率上限只有约 1.56%。这不是可接受风险的建议，而是让经营假设可以被计算；错误后果、非随机复核、误报与多轮处理均未建模。

7. 用问题与失败类型组织本地验收集

建立可版本化的问题集，并由了解业务的人记录有效答案、必要证据、适用身份与日期。包含普通查找，也包含同名文件、旧新版本、跨部门例外、资料不存在、权限不足和需要穷尽列表的问题。测试集不能只来自系统已经答对的历史对话。

评估时固定数据快照与用户身份，保留检索片段、答案和逐项判定。人工抽样检查自动评审器的误判，尤其关注少见但代价高的类别。题目或标准答案更改时留下原因；在合成基准上领先，应作为技术候选证据，而非跳过本地验收的通行证。

8. 值得部署的知识助手，会说明自己的边界

优先部署的范围可以是来源权威、版本可控、问题相对明确且有负责人接手的知识任务。对于需要跨权限拼接、频繁改变或后果重大的内容，先缩小适用范围并明确确认流程。模型升级、索引更新和权限改动都可能改变结果，应触发相应复测。

企业知识库 AI 的产品价值，不只是让用户更快看到一段答案，而是让他们更快找到可用证据、知道哪些部分不能确认，并把问题交到合适的人手里。把检索、主张、时效和责任一起交付，才更接近可靠的知识服务。