AI 与机器学习2014arxiv.org编辑精选
深度网络真的需要很深吗？
Do Deep Nets Really Need to be Deep?
进阶 · 约 35 分钟读完前置知识：神经网络基础、反向传播、模型压缩
导读
论文研究的是：深度神经网络的优势究竟来自「深度」这一结构，还是来自参数本身。作者 Ba 与 Caruana 通过模型压缩（model compression），即先用深层网络或集成模型学出函数，再训练一个单隐层的浅层网络去模仿它的输出；结果表明，在参数量相同时，浅层网络能达到与深层网络相近的精度，卷积网络也可用约 10 倍开销的浅层网络模拟。不过关键限制在于：这种浅层网络目前无法直接从原始数据训练出来。
HN 关注点：它直接挑战了「深度架构本身更具表达效率」的普遍信念，同时引出一个耐人寻味的结论：浅层网络能表达，却学不到。
后来怎么样
这篇论文是模型压缩/知识蒸馏方向的早期代表作之一，推动了将大模型能力迁移到小模型的研究；但其“深度并非必要”的强结论后来被“浅层网络难以直接从原始数据学出同等函数”的学习可行性问题所修正。
HN 评论精选
这几条评论里：支持 1 质疑 1 补充 1
支持论文证明深度架构本身并不比同等参数量的浅层网络更有表达力，深卷积网可用约 10 倍开销的浅层网模拟。
这篇论文很有意思，写得也好。他们证明，一个深层神经网络（人们原以为它的「每参数解释力」更强）可以被参数量相同的浅层网络「模拟」。Ba 和 Caruana 表明，作为架构，深度神经网络并不更强大。以前的普遍看法（至少我这么认为）是：由数据层、特征层、特征的特征层、特征的特征的特征层到预测层这样的深层结构更高效，也就是每参数解释力更强。事实并非如此。显然，一个由数据层到特征层（数量多得多）再到预测层的浅层网络，只要第一层单元数等于深层网络各层单元总数，就能取得同样精度。因此深层与浅层网络的每参数解释力大致相当。另一个结果是，深度卷积网络可以被浅层网络模拟，模拟开销约 10 倍。他们还提出一个关于「安装」ML 模型参数的类比：模型相当于硬件，参数相当于软件；运行程序包括租服务器、编译、运行，而还有一条替代路径……
ivan_ah查看英文原文
Very interesting and well written paper. They show a deep-neural network (which people thought had more explaining-power-per-parameter) can be "emulated" by a shallow neural network with the same number of parameters. What Ba and Caruana have shown is that deep neural networks---as architecture---are not more powerful. The previous general opinion (at least this is what I thought) was that deep architectures of the form: predictions layer 3 features of features of features layer 2 features of features layer 1 features data layer would be more efficient (i.e. more explaining power per parameter). Nope. Apparently, a shallow neural network of the form predictions layer 1 features (a lot more of them) data layer with an equal number of units in layer 1 as the total number of units in deep network, can obtain the same accuracy. Thus, deep neural networks and shallow neural networks have roughly equivalent explaining-power-per-parameter. A separate result shows deep convolutional neural networks can be simulated by shallow NNs with a simulation overhead of ~10x. Another cool observation they make about "installing" the parameters of ML models. We'll use the following analogy to illustrate the point: ML model <==> hardware ML model parameters <==> software Running program X involves (1) buying/renting a server, (2) compiling X for your architecture, and (3) running X. Of course there is always the alternate step (2') of installing pre-compiled binaries. Similarly, we can think of using a ML model as a three step procedure (1) ML.init() (malloc for Model.parameters), (2) Model.train() (fit parameters to training data), (3) Model.predict(new_datum). The authors point out the existence of a possibility of distributing models parameters as binaries. They demonstrate an alternate step (2') in which they train a ''FancyModel'', then set the parameters of the simple model so it "simulates" the FancyModel: ''Model.parameters = simulate(FancyModel)''. In other words, the model parameters of a ML model need not be learned through the model itself. What I find fascinating is it seems deep architectures don't have any more representational power, but they do have more learning power. Something very interesting is going on...
补充真正的发现在于：浅层网能表达该函数却学不出来，且压缩后的模型难以再增量更新。
这很有意思。论文说：「模型压缩表明，小型神经网络原则上可以学到更准确的函数；但用现有学习算法，我们无法直接在原始训练数据上训练出达到该精度的模型；只能先训练复杂集成模型，再训练神经网络去模仿它。」这是个深刻的结论，涉及「抽象」如何运作，也揭示了学到的信息如何变得越来越固化。一旦压缩了神经网络，你就无法用新数据更新这个状态少得多的压缩形式。
Animats查看英文原文
This is fascinating. "Model compression demonstrates that a small neural net could, in principle, learn the more accurate function, but with the current learning algorithms we are unable to train a model with that accuracy on the original training data; instead, we must train the complex ensemble first and then train the neural net to mimic it." That's a profound result. It bears on how "abstraction" works, and gives some insight on how learned information becomes more rigid. Once you've compressed the neural net, you can't update the compressed form, which has much less state, based on new data.
质疑质疑：结果只说明能否编码，不代表能否学习；浅层网络的学习能力会更快饱和，可能只是训练方法不够好。
这是个很有趣的结果，但和神经网络一贯的情况一样，我们必须记住：关键不在于一个模型能否被编码进另一种架构（即使熵相同），关键在它能否被学出来。用常规反向传播和 dropout 训练浅层网络时，你会发现它们的学习能力比深层网络更快「饱和」。浅层网络常常达到某点后，再加单元或加数据都没有提升；而深层网络更能利用这些额外参数（额外层）和额外训练数据。这可能是因为深层网络在概念上把学习问题「分解」成一个个增量步骤，每新增一层都是更高层的表示。但也可能只是我们还没有足够好的方法来大规模训练浅层网络——毕竟我们也是最近才学会高效训练深层网络（用自编码器或 RBM 预训练，或 Hessian-free 优化）。
fchollet查看英文原文
It's a very interesting result, but as always with neural networks we have to keep in mind that what matters is not whether a model can be encoded in a different architecture (even at equal entropy), the question is whether the model can be learned in the first place. When you work with shallow nets trained with regular backprop + dropout, you see that their learning capabilities tend to "saturate" much quicker than deep nets. Often with shallow nets, after a point you don't get better results by adding more units or more training data. But deep nets are better able to make use of these extra parameters (extra layers) and extra training data. Possibly because deep nets conceptually "break down" a learning problem into incremental steps (each new layer being a higher level of representation). But then again, maybe the problem is simply that we don't have sufficiently good methods for training shallow NNs on large-scale problems. After all, it's only recently that we figured out how to efficiently train deep nets (either pre-training with Autoencoders or RBMs, or through Hessian-free optimization).
历次讨论
- 2014Hacker News 讨论帖126 分 · 29 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选