AI 与机器学习2024nature.com编辑精选
利用语义熵检测大语言模型中的幻觉
Detecting hallucinations in large language models using semantic entropy
进阶 · 约 60 分钟读完前置知识：概率与信息熵基础、大语言模型采样机制、幻觉问题基本概念
导读
这篇 Nature 论文提出用「语义熵」（semantic entropy）检测大语言模型的幻觉：对同一问题多次采样生成回答，把语义相同的回答归为一类，再在语义层面而非字面词序列层面计算不确定性——当模型在「意思」上分歧大时，回答更可能是幻觉。作者称该方法无需标准答案，可跨数据集、跨任务使用。
HN 关注点：HN 读者一方面争论「幻觉」这个概念本身是否成立（LLM 是否只是「与真假无关」），另一方面质疑仅靠多次采样的一致性/熵能否区分「多样的正确答案」和「自信而一致的错误答案」。
HN 评论精选
这几条评论里：质疑 2 补充 2
质疑怀疑论：判断幻觉需要已知正确答案，这与使用 LLM 的目的自相矛盾。
LLM 做的唯一一件事不就是「幻觉」吗？而且要知道它是否「幻觉」，你必须事先就知道正确答案；如果你能造出一个能判断答案对错的系统，你就不需要 LLM 了。
jasonlfunk查看英文原文
Isn’t it true that the only thing that LLM’s do is “hallucinate”? The only way to know if it did “hallucinate” is to already know the correct answer. If you can make a system that knows when an answer is right or not, you no longer need the LLM!
质疑语义熵依赖未经验证的分布假设：语义多样≠错误，语义一致≠正确，可能被「自信且一致的幻觉」骗过。
大家都在争语义和拟人化，先放一边——这个方法在数学层面有真问题。对给定输入，模型对应一个输出分布；多次采样再算熵/分歧，前提是你已经知道这个分布的性质，而论文似乎没证明这个假设。比如问「汤姆·克鲁斯因何出名」，「电影明星」「Topgun」「科学教」在词向量空间相距很远、语义相似度低，但都正确；而问「泰勒·斯威夫特因何出名」，「单口喜剧」「喜剧演员」语义高度相似，却是幻觉。不看分布本身的特征，光凭向量空间的接近程度无法判断对错。
program_whiz查看英文原文
Everyone in the comments seems to be arguing over the semantics of the words and anthropomorphization of LLMs. Putting that aside, there is a real problem with this approach that lies at the mathematical level. For any given input text, there is a corresponding output text distribution (e.g. the probabilities of all words in a sequence which the model draws samples from). The approach of drawing several samples and evaluating the entropy and/or disagreement between those draws is that it relies on already knowing the properties of the output distribution. It may be legitimate that one distribution is much more uniformly random than another, which has high certainty. Its not clear to me that they have demonstrated the underlying assumption. Take for example celebrity info, "What is Tom Cruise known for?". The phrases "movie star", "katie holmes", "topgun", and "scientology" are all quite different in terms of their location in the word vector space, and would result in low semantic similarity, but are all accurate outputs. On the other hand, "What is Taylor Swift known for?" the answers "standup comedy", "comedian", and "comedy actress" are semantically similar but represent hallucinations. Without knowing the distribution characteristics (e.g multivariate moments and estimates) we couldn't say for certain these are correct merely by their proximity in vector space. As some have pointed out in this thread, knowing the correct distribution of word sequences for a given input sequence is the very job the LLM is solving, so there is no way of evaluating the output distribution to determine its correctness. There are actual statistical models to evaluate the amount of uncertainty in output from ANNs (albeit a bit limited), but they are probably not feasible at the scale of LLMs. Perhaps a layer or two could be used to create a partial estimate of uncertainty (e.g. final 2 layers), but this would be a severe truncation of overall network uncertainty. Another reason I mention this is most hallucinations I encounter are very plausible and often close to the right thing (swapping a variable name, confabulating a config key), which appear very convincing and "in sample", but are actually incorrect.
补充换一种框架：LLM 输出与真假「正交」，用「幻觉」描述本身就预设了错误的前提。
一种表述是这些是幻觉；另一种表述是这些系统「与真相正交」——它们与真假无关。有一篇论文表达了这一观点：https://link.springer.com/article/10.1007/s10676-024-09775-5
MikeGale查看英文原文
One formulation is that these are hallucinations. Another is that these systems are "orthogonal to truth". They have nothing to do with truth or falsity. One expression of that idea is in this paper: https://link.springer.com/article/10.1007/s10676-024-09775-5
补充提议把形式化验证纳入训练目标，以从根源上保证知识一致性。
当前 LLM 架构主要聚焦检索部分，学到的权重只是收敛到让下一个 token 预测最优；而把数据放进逻辑系统的能力本也应是训练目标。下一 token 预测 + 训练阶段的形式化验证 = 让模型在知识生成中保持一致，并能识别「正确的幻觉」（我称之为想象力）。流程可以是：1）用现有大模型把原训练数据转成形式化逻辑关系，生成多个解；2）用这个增强数据集训练新模型，同时输出下一个 token 和知识间的形式化关系；3）让权重优化到生成的形式化代码在证明器上取得高准确率、同时保持 token 生成准确率。我个人觉得语言是次要的，智能的基底更像是一个彼此一致的梦境式模拟，语言只是用来描述它。
badrunaway查看英文原文
Current architecture of LLMs focus mainly on the retrieval part and the weights learned are just converged to get best outcome for next token prediction. Whereas, ability to put this data into a logical system should also have been a training goal IMO. Next token prediction + Formal Verification of knowledge during training phase itself = that would give LLM ability to keep consistency in it's knowledge generation and see the right hallucinations (which I like to call imagination) The process can look like- 1. Use existing large models to convert the same previous dataset they were trained on into formal logical relationships. Let them generate multiple solutions 2. Take this enriched dataset and train a new LLM which not only outputs next token but also a the formal relationships between previous knowledge and the new generated text 3. Network can optimize weights until the generated formal code get high accuracy on proof checker along with the token generation accuracy function In my own mind I feel language is secondary - it's not the base of my intelligence. Base seems more like a dreamy simulation where things are consistent with each other and language is just what i use to describe it.
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2024Hacker News 讨论帖254 分 · 179 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选