AI 与机器学习2024arxiv.org
Differential Transformer
Differential Transformer
进阶 · 约 60 分钟读完前置知识：Transformer 架构、注意力机制、语言模型训练基础
导读
论文提出 Differential Transformer，核心是差分注意力：用两组 softmax 注意力相减来抑制无关上下文带来的注意力噪声。评论提到其报告在验证损失上 6.8B 模型可接近 11B Transformer，仅需约 62% 参数，但该机制为何有效仍有讨论。
HN 关注点：这种“噪声消除”式注意力为何有效、两组注意力如何分工，以及双倍注意力内存与参数效率之间的权衡。
HN 评论精选
这几条评论里：质疑 2 补充 2
质疑质疑差分注意力缺乏清晰的信号/噪声区分，机制不直观。
和机器学习这个新领域里的多数东西一样，我真的很困惑它为什么有效。降噪耳机的类比有帮助，但那种情况下我们清楚哪个是信号、哪个是噪声。这里如果知道的话，何必还要做降噪工作？
msoad查看英文原文
Like most things in this new world of Machine Learning, I'm really confused why this works? The analogy to noise-cancelling headphones is helpful but in that case we clearly know which is signal and which is noise. Here, if we knew why would we even bother to the noise-cancelling work?
质疑理解动机但困惑负注意力权重和平衡如何实现。
我觉得自己漏掉了关键洞察。我理解普通 softmax 注意力很难把无关内容的注意力降到零；也理解用这个相减公式可以在不产生极端离群激活的情况下把注意力权重设成精确或接近零。但它似乎也很容易产生负注意力权重（等价于对 value 向量取反后给正权重）。直觉上，要让所有不关心的东西都贴近零是很难的平衡。不过图 1 显示它确实有效，所以我不怀疑可行，只是难以想象网络具体如何做到。
Imnimo查看英文原文
I feel like I'm missing a key insight here. I understand the problem that regular softmax attention struggles to approach assigning zero attention to irrelevant stuff. And I get that having this subtraction formula makes it possible to assign exactly (or near) zero attention weight without having crazy outlier activations. But it seems like it also makes it very easy to have negative attention weight (which is equivalent to having positive attention weight on the negation of your value vectors). Intuitively, it just feels like a difficult balancing act to keep all the stuff you don't care about so close to zero. But Figure 1 clearly shows that it works, so I don't doubt that it is in fact possible. I'm just struggling to build a picture of how exactly the network accomplishes this.
补充解释两组注意力必须分化，可能一组关注信号、一组关注噪声。
我一开始没理解的关键点是：如果两组注意力学到同样的东西会怎样；因为它们的注意力 mask 相减，如果两组输出相似，整体注意力会降到零并导致高损失。所以降低损失的唯一方式是它们学会关注不同东西。最简单策略之一（论文声称它们确实如此）是一组关注相关上下文，另一组关注无关上下文；于是一组学噪声、一组学信号（并非完全泾渭分明，但我觉得是有用简化）。
iandanforth查看英文原文
The key bit I didn't understand at first was what happens if the two groups of attention learn the same thing; because their attention masks are subtracted from one another if they both output similar values the attention across the board will drop to zero and this will lead to high loss. So the only way to reduce loss is if they learn to attend to different things. One of the simplest strategies they could learn (and this paper claims that they do) is for one group to focus on relevant context and the other to focus on irrelevant context. Thus one group learns the noise and the other the signal (it's not this cut and dry but is a useful simplification for understanding IMO).
补充关注双倍注意力内存与参数效率的权衡及训练/推理差异。
如果我没理解错，这个架构用两倍注意力内存换取更高质量模型，或在相近质量下更少参数。根据拟合曲线，6.8B 的 DIFF Transformer 达到与 11B Transformer 相当的验证损失，只需要 62.2% 参数。这引出几个问题：只有 60% 参数是否会抵消双倍注意力空间，使内存占用与传统 Transformer 相近？这种权衡在训练和推理间是否明显变化？
islewis查看英文原文
> Differential attention takes the difference between two softmax attention functions to eliminate attention noise If I understand correctly, this architecture trades twice as much attention memory in exchange for either a higher quality model, or less parameters at a similar quality. > According to the fitted curves, 6.8B-size DIFF Transformer achieves a validation loss comparable to 11B-size Transformer, requiring only 62.2% of parameters This raises a few questions for me: - Would having only 60% of the parameters negate the double space for attention, leaving a similar memory profile as a traditional transformer? - Does that tradeoff change noticeably between training and inference?
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2024Hacker News 讨论帖562 分 · 177 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选