AI 与机器学习2021arxiv.org
DeepMind 用 8.7 倍更快的训练达到图像识别 SOTA
DeepMind achieves SOTA image recognition with 8.7x faster training
进阶 · 约 60 分钟读完前置知识：深度学习基础、卷积神经网络、BatchNorm
导读
这篇论文研究如何在不使用 BatchNorm 的情况下训练大规模图像识别网络，提出 Normalizer-Free 网络并结合自适应梯度裁剪等手段，在 ImageNet 上取得有竞争力的 SOTA 精度，同时据标题称训练速度提升约 8.7 倍。其核心卖点是移除长期被视为必需但麻烦的 BatchNorm，同时保持高性能。
HN 关注点：两点：去掉 BatchNorm 是否真的可行，以及性能提升到底来自新架构、归一化替代方案还是自适应梯度裁剪；此外也延伸到训练成本和 ML 研究是否过度依赖试错。
HN 评论精选
这几条评论里：支持 1 质疑 2 亲历 1
质疑训练成本过高会促使社区转向更小更快的训练；当前很多 SOTA 进展靠试错拼装而非理论理解。
我感觉 ML 社区会在取得更大突破之前，先转向更快、更小的训练，因为当 SOTA 模型需要 50 万级别的硬件成本时，很多 AI 研究根本做不起。尽管外界把 ML 研究说得很数学化，但很多工作更像炼金术而不是科学：我们并不真正理解这些神经网络为何有效。真正做高于代数水平数学的人很少，现场主要是“猜和试”式调参。很多 SOTA 模型只是把常见策略按研究者试出来最有效的方式粘在一起；普通人也许就能通过给现有模型加 RNN/GAN 层并调参来发有影响力的论文。NLP 里 roBERTa、XLNET、ELECTRA 等基本就是 BERT 的变体。我的观点是，这些并不需要多少数学或天才，主要就是不断试错。
throwaway189262查看英文原文
I have a feeling the ML community is going to pivot focus to faster and smaller training before larger advancements are made. It's simply too expensive for much AI research to happen when state of the art models take 500k of hardware to train. For all the mathematician hype around ML research, much of the work is closer to alchemy than science. We simply don't understand a great deal of why these neural nets work. The people doing math above algebra are few and the scene is dominated by "guess and check" style model tinkering. Many "state of the art models" are simply a bunch of common strategies glued together in a way researchers found worked the best (by trying a bunch of different ones). An average Joe could probably write influential ML papers by gluing RNN/GAN layers to existing models and fiddling with the parameters until they beat current state of the art. In fact, in NLP models, this is essentially what has happened with roBERTa, XLNET, ELECTRA, etc. They're all somewhat trivial variations on Google's BERT, which is more creative but yet again built on existing models. Anyways, my point is, none of this required math or genius or particularly demanding thought. It was basically let's tinker with this until we find a way that's better, using guess and check. No equations needed. We are a long way from the type of simulations done for protein folding and materials strength and basically every other scientific field. It's still the wild west
亲历JFT 预训练提升准确率反证结果可信；BatchNorm 可能被新方法替代。
我本来怀疑 ImageNet top-1 83% 那个准确率。但有人指出，模型先在 Google 专有的 1 亿图像 JFT 数据集上预训练后，准确率会升到 87% 左右。这很有意思，反而说明原结果可能是真的：如果结果是假的，JFT 预训练不该有正面效果。另一个值得说的是 AJMooch 似乎把 BatchNorm 彻底淘汰了，这很奇怪；BN 统治了约四年，但缺点明显，比如你得自己维护计数器，很烦。我们试了 evonorm-s，第一次就完美工作，没有损失通用性，基本可直接替换。
sillysaurusx查看英文原文
I was skeptical of the 83%-whatever top-1 accuracy on Imagenet. But someone pointed out that the accuracy increases when the model is pretrained on JFT, google's proprietary 100-million image dataset, the model's accuracy increases to 87%-whatever. That's pretty interesting. It implies that the original accuracy rating might be legit. The concern is that we're chasing the imagenet benchmark as if it's the holy grail, when in fact it's a very narrow slice of what we normally care about as ML researchers. However, the fact that pretraining on JFT increases the accuracy means that the model is generalizing, which is very interesting; it implies that models might be "just that good now." Or more succinctly, if the result was bogus, you'd expect JFT pretraining to have no effect whatsoever (or a negative effect). But it has a positive result. The other thing worth mentioning is that AJMooch seems to have killed batch normalization dead, which is very strange to think about. BN has had a long reign of some ~4 years, but the drawbacks are significant: you have to maintain counters yourself, for example, which was quite annoying. It always seemed like a neural net ought to be able to learn what BN forces you to keep track of. And AJMooch et al seem to prove this is true. I recommend giving evonorm-s a try; it worked perfectly for us the first time, with no loss in generality, and it's basically a copy-paste replacement. (Our BigGAN-Deep model is so good that I doubt you can tell the difference vs the official model. It uses AJMooch's evonorm-s rather than batchnorm: [1] https://i.imgur.com/sfGVbuq.png [2] https://i.imgur.com/JMJ1Ll0.png and lol at the fake speedometer.)
支持最大亮点是移除长期必需但不受欢迎的 BatchNorm。
这里最重要的是移除了 BatchNorm。人们出于各种理论和实践原因一直不太喜欢 BatchNorm，但它又是所有顶尖模型所必需的。如果这能让我们永久摆脱它，那会很好。
modeless查看英文原文
The big deal here is the removal of BatchNorm. People never really liked BatchNorm for various theoretical and practical reasons, and yet it was required for all the top performing models. If this allows us to get rid of it forever that will be really nice.
质疑质疑性能提升归因：难以区分新架构与自适应梯度裁剪的贡献。
Yannic Kilcher 已经出了很好的视频，其中有一些批判性思考。他提到的一点是，作者同时引入了替代 BatchNorm 的新方法和新架构；组合效果很好，但很难分清收益有多少来自新架构，多少来自他们提出的 adaptive gradient clipping。
Yenrabbit查看英文原文
Yannic Kilcher has a great video on this out already, including some thoughtful critique. https://youtu.be/rNkHjZtH0RQ One of the things he mentioned was that they introduce this fancy alternative to BatchNorm AND come up with a fancy new architecture. The combination does really well, but it isn't clear how much of the gain is due to the new improved architecture Vs the 'adaptive gradient clipping' they introduce.
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2021Hacker News 讨论帖11 分 · 3 评
- 2021Hacker News 讨论帖291 分 · 83 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选