AI 与机器学习2024arxiv.org
Mixture-of-Depths：在 Transformer 中动态分配计算量
Mixture-of-Depths: Dynamically allocating compute in transformers
进阶 · 约 60 分钟读完前置知识：Transformer 架构、残差连接、Mixture-of-Experts 基础
导读
这篇论文（DeepMind）提出 Mixture-of-Depths：在 Transformer 中给每个 block 加一个路由器，决定哪些 token 真正经过该 block、哪些直接跳过（残差直连），并用 top-k 容量上限把总计算量固定在和基线模型相同的水平。这样模型可以在 token 维度上动态分配算力，在同等 FLOPs 下取得与基线相当甚至更好的效果。
HN 关注点：「不再对每个 token 花同样的算力」这一思路，以及它的路由是否真的学到了有意义的东西、还是只是变相 dropout。
HN 评论精选
这几条评论里：支持 1 质疑 1 补充 2
支持高度评价该机制，但对因果路由如此容易学到感到意外。
2024 年最重要的论文。让模型不必对每个 token 使用相同算力的想法已经存在一段时间了，但这是我见过的第一个有说服力的实现机制。另外让我意外的是：路由器只依赖因果信息（不看未来 token）就能达到 99% 的准确率，这似乎太容易了，有人也觉得惊讶吗？
nl查看英文原文
Most important paper of 2024. The idea that we want models not to have to use the same amount of compute for every token has been around for a while. This is the first compelling mechanism I've seen for doing it. > Equipped with these new methods, we can sample autoregressively by choosing to route tokens to or around a block based on the router’s output, which does not depend on any information from future tokens. We provide empirical evidence that this is a relatively easy auxiliary task that quickly achieves 99% accuracy. Does anyone else find this is a bit surprising?
质疑质疑论文的实际贡献被夸大，认为更接近 dropout。
我觉得摘要和正文对不上。它并不是真的给某些序列分配更多算力，本质上只是引入了某种 dropout。也许是一体两面，但读起来还是有点奇怪。
maxrumpf查看英文原文
The abstract and the rest of the paper don't really match imo. It's not really allocating more to some sequences, but just introducing ~dropout. Might be different sides to the same coin, but was still a weird read.
补充认为递归路由与模型内部思维链是下一步方向。
更复杂的路由肯定会越来越普遍。具体来说，我认为早晚会走向递归路由，也就是让 token 再次经过同一组专家。未来「思维链」会在模型内部递归地发生。
whimsicalism查看英文原文
I think more complicated routing is absolutely going to become more common. Specifically, I think at some point we are going to move to recursive routing, ie. pass back through a set of experts again. In the future, 'chain-of-thought' will happen internal to the model recursively
补充用预算和注意力分配的比喻通俗解释 MoD。
简化版导读：想象一个助手通常对你说的每个词都投入同等的注意力。现在我们教它学会分配「脑力」：更关注对理解语义最重要的词，并且能根据上下文动态调整。为了不让它过载，我们给它设一个总预算，它只能在固定数量的词上花脑力，于是必须自行决定哪些词最重要。即便如此，它在预算内仍然灵活，可以某些词多花、某些词少花。这样训练出来的助手既学会了智能地聚焦，效率也很高。
panqueca查看英文原文
Simplified Intro Version: Imagine you have a smart assistant that can understand and process the words you say to it. Usually, this assistant pays equal attention to every word you say, no matter how important or unimportant each word is to the overall meaning of your message. Now, imagine that we found a way to teach the assistant to be smarter about how it uses its "brain power." Instead of giving equal attention to every word, the assistant learns to focus more on the words that are most important for understanding what you mean. It can even adjust this focus on the fly, paying more attention to different words depending on the context of your message. To make sure the assistant doesn't get overwhelmed, we also set a limit on how much total "brain power" it can use at any given time. It's like giving the assistant a budget and saying, "You can only spend your brain power on a certain number of words at a time." The assistant then has to decide which words are most important to focus on. Even with this limit, the assistant is still flexible in how it uses its brain power. It might spend more on certain words and less on others, depending on what you're saying. This means that while we always know the total amount of brain power the assistant is using, it can adapt to different situations and prioritize what's most important. When we teach the assistant using this method, it not only learns to focus its attention intelligently but also does so very efficiently. It can understand you just as well as an assistant that pays equal attention to every word, but it uses less brain power overall. This makes the assistant much faster at responding to you and processing new information.
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2024Hacker News 讨论帖281 分 · 83 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选