AI 与机器学习2020arxiv.org编辑精选
深度学习的计算极限
The Computational Limits of Deep Learning
进阶 · 约 50 分钟读完前置知识：深度学习基础、算力与硬件性能、趋势外推分析
导读
这篇论文研究深度学习对计算资源的需求是否可持续，指出模型训练所需的算力增长远快于硬件性能提升，存在明显的“计算极限”。作者通过分析大量模型数据，讨论仅靠扩大算力难以长期维持，需要算法、硬件和理论层面的协同创新。
HN 关注点：深度学习是否撞上算力墙，以及“暴力堆算力”与算法、因果推断、理论突破之间的路线之争。
HN 评论精选
这几条评论里：支持 2 补充 2
支持当前 ML 多是暴力统计拟合，算力决定适用边界，复杂领域终会撞墙。
我对当前 ML 方案的理解是，它们大多是暴力统计模型，希望碰上一个令人满意的 p 值（有些实现只是暴力得更优雅）。算力决定了你能把方案应用到多大的有限领域。一个简化类比是鸟群行为：当前 ML 需要大量鸟群飞行/非飞行数据，并调校暴力模型的 p 值来预测是否群集；但鸟群行为几十年来用个体简单规则和群体涌现现象就能轻松建模。我担心多数 ML 只是在摘暴力方法的低垂果实，最终会撞墙，停滞在蠕虫等生物能解决的“简单”问题水平，因为许多领域的复杂度超过任何现实可想象的简单数学算力。
fallous查看英文原文
My interpretation of current ML solutions is that it's mostly brute force statistical models hoping to hit a satisfactory p-value (some implementations being more elegant in the brute force than others). The calculating power defines the limits of the finite domain you can apply the solution towards. A simplified analogy that I believe is applicable is the application of flocking behavior in birds. Current ML implementations would demand a large dataset of groups of birds in flight, both flocking and non-flocking, and curating the correct p-value of brute-forced models that allege to predict whether behavior is flocking or not (and in the case of an individual bird whether it is appropriate flocking behavior given the individual behaviors of birds in the dataset and their circumstances). But flocking behavior is easily modeled right now, and has been for decades, using simple rules for individuals and depending upon emergent phenomena within a group. I'm concerned that most of the ML efforts now are merely attaining the low-hanging fruits of brute force but will run into a wall that halts progress at the level of relatively "easy" things solved by worms and other comparable biological solutions since so many domains have a level of complexity that would exceed any realistically imaginable level of simple mathematical computing power.
补充算力需求会转化为高效硬件研发的经济激励。
我们可以对深度学习不断增长的算力需求感到绝望，也可以把它视作经济激励：开发更强大、能效更高、边际成本更低且更可持续的硬件。换句话说，深度学习的算力需求似乎已推动基础研究寻找更绿色、便宜、高效的硬件。经济激励非常强：谁能以最低边际成本驾驭最多算力，谁就能赢得市场。PS：比特币挖矿也类似，同样在推动更绿色、便宜、高效、强大的硬件研发。
cs702查看英文原文
We can look Deep Learning's growing demands for computation and despair, or view those growing demands as an economic incentive to develop more powerful hardware that uses energy more efficiently at a lower marginal cost and in a more sustainable manner. In other words, Deep Learning's growing need for computing power seems to have reached a point at which it is now motivating fundamental research to find greener, cheaper, more energy-efficient hardware. The economic incentives are very powerful: Whichever companies (or organizations, or countries) find ways to harness the most computing power at the lowest marginal cost will win the race in this market. -- PS. The same could be said for Bitcoin mining: it is also motivating fundamental research to develop greener, cheaper, more energy-efficient, more powerful hardware. Whoever finds ways to harness the most computing power at the lowest marginal cost will make the most money processing transactions on the network.
支持深度学习无法区分因果与相关，堆算力收益递减，需要因果算法。
读这篇论文让我想起 AI 持续演进核心算法有多重要。深度学习模型本质上是模式匹配机器，无法区分因果与相关。随着我们投入更多算力和资金，深度学习会因此出现性能收益递减。要实现 AGI 这一 AI 的圣杯，我们需要能识别因果的算法。Judea Pearl 的《The Book of Why》很好地说明了这一点。深度学习是巨大进步，我们才刚开始看到其影响，但它不足以实现 AI 最雄心勃勃的目标。
peterthehacker查看英文原文
Reading this paper reminds me how important it is for AI to continue to evolve it’s core algorithms. Deep learning models are effectively pattern matching machines that cannot separate causality from correlation. As we throw more compute and $$$ at deep learning models we will experience diminishing returns in performance because of this. For us to achieve AGI[0], the holy grail of AI, we will need to develop algorithms that can recognize causality somehow. Judea Pearl’s “The Book of Why”[1] does a great job articulating why this is important. Deep learning is a big leap forward and we’re only beginning to see its impacts, but it’s not sufficient to achieve AI’s most ambitious goals. [0] https://en.m.wikipedia.org/wiki/Artificial_general_intelligence [1] https://www.amazon.com/Book-Why-Science-Cause-Effect/dp/046509760X
补充类似 90 年代整数规划，理论/算法突破比单纯算力增长更能带来跃升。
当前领域状态让人想起 90 年代末的离散数学优化。当时算力快速增长，但我们求解超过一百个二进制变量问题的能力几乎没提升。直到整数规划的新理论成果进入求解算法，性能才出现阶梯式提升。
whatever1查看英文原文
The current state of the field is reminiscent of discrete mathematical optimization in the late nineties. The computational power was increasing rapidly back then but our our ability to solve problems with more than a hundred of binaries hadn’t improve almost at all. It was only after new theoretical results in integer programming found their way into the solution algorithms when we saw a stepwise increase of performance.
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2020Hacker News 讨论帖16 分 · 0 评
- 2020Hacker News 讨论帖168 分 · 73 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选