AI 与机器学习2025arxiv.org
思维链可监控性：AI 安全中一个新的脆弱机会
Chain of thought monitorability: A new and fragile opportunity for AI safety
进阶 · 约 40 分钟读完前置知识：大语言模型基础、思维链（CoT）推理、AI 安全与对齐基础
导读
这是一篇多名作者合著的立场/共识型论文：它指出，当前推理模型用人类可读的语言“思考”，其思维链（CoT）可被外部监控，从而构成 AI 安全上一个独特而脆弱的机会；作者主张前沿开发者应尽量保留这种可监控性，并配合多层监控机制。论文的核心是提出倡议、分析可监控性可能被侵蚀的风险，而非提出新的技术方法。
HN 关注点：的焦点是这种可监控性注定短命：竞争压力会推动模型改用人类不可读的隐空间推理，而且公开研究监控方法本身也会削弱它的效果。
HN 评论精选
这几条评论里：质疑 4
质疑追问：模型转向非人类语言的“neuralese”推理这一预测是否可信。
论文说，AI 系统用人类语言“思考”为 AI 安全提供了独特机会：我们可以监控它们的思维链（CoT），以发现其意图不端。而 AI2027 预测，一旦我们找到替代“人类语言”思考的方式，LLM 的性能就会提升——至少那个视频给我这种印象，我想这就是“neuralese”所指的东西。这个预测可信吗？
barbazoo查看英文原文
> AI systems that "think" in human language offer a unique opportunity for AI safety: we can monitor their chains of thought (CoT) for the intent to misbehave. AI2027 predicts a future in which LLM performance will increase once we find alternatives to thinking in "human language". At least the video gave me that impression and I think this is what "neuralese" is referring to. Is that a credible prediction?
质疑公开研究监控方法本身会让模型（或其训练者）学会规避，从而削弱监控效果。
讽刺的是，每发表一篇关于监控思维链的论文，都会降低这项技术对强大 AI 模型有效的可能性。
alach11查看英文原文
Ironically, every paper published about monitoring chain-of-thought reduces the likelihood of this technique being effective against strong AI models.
质疑引用隐空间推理的论文，认为竞争压力会让可读 CoT 迅速过时。
不幸的是，我认为追求模型性能的竞争压力会让这种可监控性维持不了多久，用英文文本推理不太可能是最优方案。研究者已经在往这个方向推进了：有工作提出在隐空间中隐式推理、通过迭代循环块在测试时扩展到任意深度，不需要专门训练数据、可用小上下文窗口，还能捕捉难以用文字表达的推理类型；还有论文研究在连续隐空间而非语言空间做推理。如果思维链不再是人类可读的格式，就没法监控了。
vonneumannstan查看英文原文
Unfortunately I think the competitive pressures to improve model performance makes this kind of monitor-ability is short-lived. It's just not likely that textual reasoning in english ismost optimal. Researchers are already pushing in this direction: https://arxiv.org/abs/2502.05171 "We study a novel language model architecture that is capable of scaling test-time computation by implicitly reasoning in latent space. Our model works by iterating a recurrent block, thereby unrolling to arbitrary depth at test-time. This stands in contrast to mainstream reasoning models that scale up compute by producing more tokens. Unlike approaches based on chain-of-thought, our approach does not require any specialized training data, can work with small context windows, and can capture types of reasoning that are not easily represented in words. We scale a proof-of-concept model to 3.5 billion parameters and 800 billion tokens. We show that the resulting model can improve its performance on reasoning benchmarks, sometimes dramatically, up to a computation load equivalent to 50 billion parameters." https://arxiv.org/abs/2412.06769 Can't monitor the chain of thought if it's no longer in a human legible format.
质疑质疑 40 多位作者的贡献与论文的新颖性：是否只是提出一个倡议。
我有点困惑这 40 位作者到底贡献了什么。论文似乎只是提出一个建议——为安全而监控思维链。这算新颖之处吗？还是说一个人提出想法，40 多人表示同意，然后就被列进作者名单了？
rdtsc查看英文原文
> Thus, safety measures for future AI agents will likely need to employ multiple monitoring layers that hopefully have uncorrelated failure modes [...] We encourage the research community and frontier AI developers to make best use of CoT monitorability and study how it can be preserved. I am bit confused what all the 40 authors contributed to here. The paper seems to make a suggestion - monitor the chain of thought for safety. Is that the novelty part? But then, did one person come up with the idea and all 40+ people agreed to it and there put in the authors list.
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2025Hacker News 讨论帖134 分 · 64 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选