AI 与机器学习2016arxiv.org
超级智能无法被约束：来自可计算性理论的教训
Superintelligence cannot be contained: Lessons from Computability Theory
进阶 · 约 60 分钟读完前置知识：可计算性理论、停机问题与 Rice 定理、AI 安全基本概念
导读
这篇论文从可计算性理论出发，论证“超级智能无法被完全遏制”：它假设超级智能程序包含通用图灵机可执行的所有程序，进而借助停机问题和 Rice 定理证明，判定某程序是否会对人类造成伤害（Harm 函数）是不可判定的，因此无法设计出绝对安全的遏制机制。
HN 关注点：HN 读者围绕其核心假设是否合理展开争论：用停机问题论证“不可能”是否成立，以及“超级智能=可计算世界一切”的定义是否过于神化。
HN 评论精选
这几条评论里：质疑 5 补充 1
质疑讽刺该论文的方法是套用停机问题批量“证明”任何概念不可计算的模板。
我承认自己是个失意的学者，但偶尔读到这样的论文，会觉得这种失意也许没什么道理。按这篇论文的套路，证明任何东西不可计算都可以这样：第一步，假设存在算法 Beauty(R,D)，给定程序 R 和输入 D，判断是否“美”；第二步，构造一个对该算法的停机问题包装的高阶函数；定理：美是不可判定的。证明：反证，若停机问题可判定，则第二步总有答案，但停机问题不可判定，矛盾。把“美”换成“棒”“善”“恐惧”等词即可批量生产定理。
cambalache查看英文原文
I confess I am a frustrated academic, but from time to time I read a paper like this one and and convince myself that my frustration may perhaps be unwarranted. RECIPE TO PROVE ANYTHING IS INCOMPUTABLE (According to this paper). Example, beauty is incomputable Step 1. Assume there is an algorithm Beauty(R,D) that given the program R and the input D, will return True if it is beautiful and False otherwise. Step 2. Create a high-order function that is a simple wrapper of the halting problem over the algorithm created in step 1 Theorem. Beauty is undecidable. Proof. Assume by contradiction that the halting problem is decidable, then, feed it the function created in step 2, thus you can always have an answer for step 1, but since the halting problem is undecidable this is not possible. For more variety please substitute beauty for "awesomeness", "arachnid power", "goodness","fear" and so on.
质疑质疑用停机问题论证不可能性：现实系统状态有限，不适用。
凡是借助停机问题论证某事不可能的说法，一般都是靠不住的。停机问题只适用于状态无限的确定性系统；如果状态有限且确定，系统最终必然重复某个状态或停机。注意“非常大”和“无限”在这里不是一回事。由于组合爆炸或噪声而无法预测，才是更有说服力的论证方式。
Animats查看英文原文
Arguments that claim something is impossible from an argument related to the halting problem are generally bogus. The halting problem applies only to deterministic systems with infinite state. If you have finite state and determinism, you have to eventually repeat a state, or halt. Note that "very large" and "infinite" are not the same thing here. Not being able to predict something for combinatorial or noise reasons is a more credible argument.
质疑认为论文脱离现实，超级智能本身能否实现都未知，讨论遏制为时过早。
这基本就是科幻。考虑到当前技术水平与假想中的超级智能相距甚远，在引言里提“近期进展”有些误导。我们没有超级智能，也不知道如何着手创造它，很可能连正确的硬件是什么样都不知道，甚至不知道它是否可能实现。
st1x7查看英文原文
This is just science fiction. To mention "recent developments" in the introduction is somewhat misleading considering how far the current state of technology is from their hypothetical superintelligence. We don't have superintelligence, we don't have the remote idea of how to get started on creating it, in all likelihood we don't even have the correct hardware for it or any idea what the correct hardware would look like. We also don't know whether it's achievable at all.
质疑指出核心假设把超级智能神化为可计算一切，不符合现实中的 AGI。
论文有一个很大且相当不合理的假设：“假设超级智能将包含一个程序，涵盖通用图灵机在输入复杂到世界状态级别时可执行的所有程序”。这实际上是把“超级智能”定义为“神一般的智能”（能计算关于世界的一切），而第一个明显比人类聪明的 AGI 绝不会长这样。
argonaut查看英文原文
This seems like a very big, pretty unreasonable assumption: "Assuming that a superintelligence will contain a program that includes all the programs that can be executed by a universal Turing machine on input potentially as complex as the state of the world" This assumption is essentially assuming that "superintelligence" means "god-like intelligence" (god-like because it can compute anything about the world), which is not what the first AGIs that are even significantly smarter than humans would look like.
补充认为用不可判定性建模安全问题是错的，实践中应像法律一样按具体案例积累规则。
有趣的文章。虽然有些水分，但核心观点——一般性的 Harm(R,D) 函数等价于停机问题——一旦问出这个问题就几乎是显然的（机器人三定律也就这样了）。我的疑问是这种建模方式是否正确：人类自己都难以定义幸福、伤害等概念，AI 未必做得更好。人类成功的减害体系往往是针对具体情形逐步积累规则，而非从一般原理演绎，各法律体系都是如此且运转良好。所以也许我们终究还是需要律师？
hodgesrm查看英文原文
Fun post. There's some fluff on the way but the central idea that the general function Harm(R, D) is equivalent to the halting problem seems almost trivially obvious once you ask the question. (So much for the laws of robotics.) My question is whether this approach even models the issue correctly. We humans have problems just defining what terms like happiness, harm, and other general concepts even mean. There's no reason to think that AIs will be any better, whether assisted by us or through their own methods of discovery. Many successful systems humans have developed for reducing harm take the approach of building up rules for specific cases rather than trying to deduce the cases and their solutions from general principles. Every legal system I can think of works this way and they have by and large worked pretty well. The successful ones have a method for adding to case law in a systematic way, ideally without going to war in the process. So maybe we should keep the lawyers around after all?
质疑用编译器类比反驳：不可判定不排除工程上做到“足够好”的安全。
加入时间因素并换个定义会更合理：“假设超级智能包含一个能高效推断/演绎出通用图灵机可执行程序的程序”。编译器也常与停机问题打交道：求最优解就是变相的停机问题，但编译器采取务实方案，虽不完美却可以非常好。类比到超级智能：我们也许造不出永不造成伤害的完美机器，但这不妨碍它做得非常非常好。
leblancfg查看英文原文
Let's throw in a time factor here, and change the way we define our machine: "Assuming that a superintelligence [SI] will contain a program that can efficiently infer/deduce all the programs that can be executed by a universal Turing machine on input potentially as complex as the state of the world". This seems much more reasonable in terms of how thinking gets done. We see links to the Halting Problem pop up at times when discussing compilers. Now, compilers try to interpret code in the most perfect way possible. But of course, finding the optimal solution is just the Halting Problem in disguise, and instead they opt for a more pragmatic solution. While none are perfect, some of them are also really, really good, though (http://lemire.me/blog/2016/05/23/the-surprising-cleverness-of-modern-compilers/). Now this is just an analogy, but I think the parallel can be drawn back to the SI scenario. We might not have SIs that never cause harm perfectly, but this does not preclude them from being really, really good at it. Because if you admit the definition above, you're also implying that it will efficiently infer/deduce its harm function.
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2016Hacker News 讨论帖29 分 · 14 评
- 2021Hacker News 讨论帖157 分 · 222 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选