AI 与机器学习2024arxiv.org
超越标准多头注意力的新注意力机制
New attention mechanisms that outperform standard multi-head attention
入门 · 约 45 分钟读完前置知识：Transformer 基础、多头注意力机制
导读
论文《You Need to Pay Better Attention》提出三种新的注意力变体：Optimised Attention（参数为标准的 3/4、每个 head 少一次矩阵乘法）、Efficient Attention（参数减半、少两次矩阵乘法、最高快一倍）和 Super Attention（在视觉与 NLP 任务上超过标准注意力且更省参数）。实验在 MNIST、CIFAR100、IMDB 影评和亚马逊评论等数据集上进行，属于对标准多头注意力的轻量化改进尝试。
HN 关注点：HN 读者一方面对“超越标准注意力”的说法持怀疑态度，质疑这些小数据集上的增益能否扩展到大规模 LLM；另一方面借机讨论为何 RWKV、Mamba 等替代方案至今仍难进生产。
HN 评论精选
这几条评论里：质疑 3 补充 1
质疑质疑 Transformer 的静态、固定计算路径，追问更动态、带工作记忆的架构。
谁能给我指几个看起来真有可能通向 AGI 的模型？我大致理解 Transformer 架构：把一个滑动窗口里的 X 个 token 乘上一组静态矩阵，产出 1 个新 token——这不像通往 AGI 的路。是的，复杂的特征提取很惊艳，但有没有更动态、有工作记忆、执行路径不那么受限的模型？
jawon查看英文原文
Can anyone point me to models that look like they might actually be useful in moving towards AGI? I feel like I have a basic understanding of the transformer architecture, and multiplying X tokens in a sliding window across a set of static matrices to produce 1 new token does not look like a path to AGI. Yes, the complex feature extraction is impressive. But are there any models that, I don't know, are more dynamic? Have a working memory? Have a less limited execution path?
质疑怀疑新方法能落地：资本充足却仍无人用非 Transformer 做生产级 LLM，说明问题不在惯性。
表示怀疑。除了老牌 Transformer，还有 RWKV、Mamba 等一堆替代方案，可我们至今仍在用 Transformer（其实只是 decoder 部分）。是因为产业惯性太大、不愿采用新方法吗？我不这么认为——这个市场有几十亿美元，人人都想分 AI 蛋糕，没道理忽视有前景的方法。那为什么如今几乎看不到非 Transformer 的生产级 LLM？
behnamoh查看英文原文
Pressing [X] to doubt. There are many alternatives to the good old transformers: RWKV, Mamba, etc. Yet here we are, still using transformers (actually, just the decoder part). Is it because the industry has so much inertia to pick up new methods? I doubt it because there's $BILLIONS in this market and everyone wants a piece of the AI cake, so it doesn't make sense to ignore promising methods. Why, then, we barely see any non-transformer production-ready LLM these days?
质疑指出实验只在 MNIST/CIFAR/IMDB 等小规模数据上，怀疑无法扩展到 LLM 规模。
“我们在 MNIST、CIFAR100、IMDB 影评和亚马逊评论数据集上评估了所提出的注意力机制。”听起来很棒，但我不指望它能扩展。
317070查看英文原文
> we evaluate the presented attention mechanisms on MNIST, CIFAR100, IMDB Movie Reviews, and Amazon Reviews datasets. It sounds amazing, but I'm not holding my breath this one will scale.
补充补充原文关键数据，并推测 FFT/量子傅里叶变换可替代注意力。
既然自注意力可以用 FFT 替代（代价是精度下降、但能省电），我猜量子傅里叶变换也能在 LLM 中替代注意力。并引用了原文摘要：Optimised Attention 参数为 3/4、每 head 少一次矩阵乘法；Efficient Attention 参数减半、少两次矩阵乘法、最高快一倍；Super Attention 在视觉和 NLP 任务上显著超越标准注意力且参数和矩阵乘法更少。另引 Infini-attention 的压缩记忆工作作补充。
westurner查看英文原文
Because self-attention can be replaced with FFT for a loss in accuracy and a reduction in kWh [1], I suspect that the Quantum Fourier Transform can also be substituted for attention in LLMs. [1] https://syncedreview.com/2021/05/14/deepmind-podracer-tpu-based-rl-frameworks-deliver-exceptional-performance-at-low-cost-19/ "You Need to Pay Better Attention" (2024) https://arxiv.org/abs/2403.01643 : > Our first contribution is Optimised Attention, which performs similarly to standard attention, but has 3/4 as many parameters and one matrix multiplication fewer per head. Next, we introduce Efficient Attention, which performs on par with standard attention with only 1/2 as many parameters as many parameters and two matrix multiplications fewer per head and is up to twice as fast as standard attention. Lastly, we introduce Super Attention, which surpasses standard attention by a significant margin in both vision and natural language processing tasks while having fewer parameters and matrix multiplications. "Leave No Context Behind: Efficient Infinite Context Transformers" (2024) https://arxiv.org/abs/2404.07143 : > A key component in our proposed approach is a new attention technique dubbed Infini-attention. The Infini-attention incorporates a compressive memory into the vanilla attention mechanism and builds in both masked local attention and long-term linear attention mechanisms in a single Transformer block. We demonstrate the effectiveness of our approach on long-context language modeling benchmarks, 1M sequence length passkey context block retrieval and 500K length book summarization tasks with 1B and 8B LLMs. Our approach introduces minimal bounded memory parameters and enables fast streaming inference for LLMs.
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2024Hacker News 讨论帖233 分 · 49 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选