AI 与机器学习2025arxiv.org
大型语言模型面前的墙
The wall confronting large language models
进阶 · 约 45 分钟读完前置知识：机器学习基础、数值分析、LLM 基础
导读
这篇论文讨论大语言模型在继续扩展时所面对的根本性“高墙”：作者从浮点精度有限、离散导数与连续数学之间的鸿沟，以及训练和推理的巨大能耗等角度，论证单纯扩大规模未必能通向可靠推理。文中还把 LLM 的能耗与人脑约 20 瓦的功耗作对比，并强调概率化序列生成缺少回溯等符号推理能力。评论者普遍认为文章论证松散、缺少具体算例，对其核心主张存疑。
HN 关注点：论文把“规模化会遇到极限”这一老争议，重新放到浮点误差、能耗和推理可靠性上讨论，但行文与论证质量也成了讨论焦点。
HN 评论精选
这几条评论里：支持 1 质疑 3
支持LLM 本质上是概率计算，无法替代符号推理与回溯。
马尔可夫链与大语言模型在形式上具有外延等价性，但似乎只有 Gary Marcus 在谈这一点。他一直强调，无论图（graph，即模型结构）多大，符号理解都无法还原为概率计算，模型仍然缺少回溯这类基础能力（比如 Prolog 这类编程语言就有）。我认为 Gary 基本全对。概率生成模型很好玩，但再多的概率序列生成也替代不了逻辑推理。
measurablefunc查看英文原文
There is a formal extensional equivalence between Markov chains & LLMs but the only person who seems to be saying anything about this is Gary Marcus. He is constantly making the point that symbolic understanding can not be reduced to a probabilistic computation regardless of how large the graph gets it will still be missing basic stuff like backtracking (which is available in programming languages like Prolog). I think that Gary is right on basically all counts. Probabilistic generative models are fun but no amount of probabilistic sequence generation can be a substitute for logical reasoning.
质疑作者的人脑与 LLM 能耗对比不公平，忽略了训练数据规模与服务规模。
Sauro，如果你看到这条：请别再写这种内容空洞的推测了——『顺便一提，这距离人脑约 10^15 个神经连接还差三个数量级，却消耗了约一亿倍的功率（以 GW 计，而人脑只需区区 20 瓦）』。就算一个人从 30 万年前人类出现起每天读八小时书，也没有哪个大脑有时间读完现代 LLM 一次训练所用的全部材料。更关键的是，LLM 的推理在能效上远高于人类推理（可参考 B200 解码 671B 参数模型的能耗，再估算人类写一本书的信息量作为更大批次的一部分所需的能量）。推理能耗高主要是因为我们在用同一个模型服务数亿人，人类没有这种规模能力。
pama查看英文原文
Sauro, if you read this, please refrain from such low-content speculative statements: “On a loose but telling note, this is still three decades short of the number of neural connections in the human brain, 1015, and yet they consume some one hundred million times more power (GWatts as compared to the very modest 20 Watts required by our brains).” No human brain could have time to read all the materials of a modern LLM training run even if they lived and read eight hours a day since humans first appeared over 300,000 years ago. More to the point, inference of an LLM is way more energy efficient than human inference (see the energy costs of a B200 decoding a 671B parameter model and estimate the energy needed to write the equivalent of a human book worth of information as part of a larger batch). The main reason for the large energy costs of inference is that we are serving hundreds of millions of people with the same model. No humans have this type of scaling capability.
质疑作者对浮点精度与导数离散化的论证有误，且与 LLM 的实际优化关系不大。
那篇文章很奇怪，似乎对核反应堆念念不忘，还误解了浮点运算的工作方式。中学就学过：连续导数是离散版本在位移 h 趋于零时的极限。如果计算机能有无限精度，这句话在实践上和连续数学里一样成立；但没有计算机负担得起无限精度，标准 IEEE 双精度浮点的精度约为第 16 位，小于 10^-16 的数基本就是纯噪声。因此当 h 小于机器精度时，舍入误差压过离散化误差，离散导数就开始偏离连续值。没错，对数据求导有噪声问题，梯度跟随者有时会卡住，低通滤波可以平滑数据、降低导数噪声。但这跟 LLM 有关系吗？机器学习优化的一个重要洞见是：在高维空间里通常总有某个维度带有显著信号，能帮你跳出局部极小值。大多数机器学习都处在这种高维但低分辨率的数据点上。
Animats查看英文原文
That article is weird. They seem obsessed with nuclear reactors. Also, they misunderstand how floating point works. As one learns at high school, the continuous derivative is the limit of the discrete version as the displacement h is sent to zero. If our computers could afford infinite precision, this statement would be equally good in practice as it is in continuum mathematics. But no computer can afford infinite precision, in fact, the standard double-precision IEEE representation of floating numbers offers an accuracy around the 16th digit, meaning that numbers below 10−16 are basically treated as pure noise. This means that upon sending the displacement h below machine precision, the discrete derivatives start to diverge from the continuum value as roundoff errors then dominate the discretization errors. Yes, differentiating data has a noise problem. This is where gradient followers sometimes get stuck. A low pass filter can help by smoothing the data so the derivatives are less noisy. But is that relevant to LLMs? A big insight in machine learning optimization was that, in a high dimensional space, there's usually some dimension with a significant signal, which gets you out of local minima. Most machine learning is in high dimensional spaces but with low resolution data points.
质疑论文可读性差、缺少具体算例，难以理解其论点。
这篇论文很难读：没有一个完整走通的具体算例，文字浮夸，公式也帮不上忙。我完全抓不住这篇论文的要点。
Scene_Cast2查看英文原文
The paper is hard to read. There is no concrete worked-through example, the prose is over the top, and the equations don't really help. I can't make head or tail of this paper.
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2025Hacker News 讨论帖172 分 · 200 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选