AI 与机器学习2019arxiv.org
单头注意力 RNN
Single Headed Attention RNN
进阶 · 约 60 分钟读完前置知识：循环神经网络、注意力机制、语言建模基础
导读
这篇论文提出 SHA-RNN（单头注意力 RNN），作者 Stephen Merity 探讨在语言建模任务中是否真的需要多头注意力和大规模算力，用单 GPU 即可训练出在 enwik8 等基准上具有竞争力的模型。论文以非正式、幽默的写作风格写成，强调可复现性和简单设计。
HN 关注点：既因为作者质疑了'堆算力刷榜'的深度学习研究风气，也因为其类似 YOLOv3 的非正式写作风格和单 GPU 可复现性引发讨论。
HN 评论精选
这几条评论里：支持 1 质疑 1 补充 1 亲历 1
质疑质疑实验设置与命名矛盾，并指出缺少具体的速度对比数据
写作风格很有趣。初看几点笔记：实验里他实际上把单头注意力模型用在了 4 个头上，这和名字有点矛盾吧？主要动机是性能（主要是训练速度），所以对比中给出具体的训练时间会更好。他提到 Adaptive Transformer 也能在单 GPU 上几小时内训练完，而且在对比中 Adaptive Transformer 的 BPC（enwik8）好得多，参数还略少。那 Adaptive Transformer 不是在速度和 BPC 上都更优吗？还是说论证的是推理速度？那应该比较推理速度才对。
albertzeyer查看英文原文
The writing style is amusing. :) Some notes from a first glance: * In the experiments, I see that he uses the Single Headed Attention model actually also with 4 heads, which is kind of a contradiction to the name, isn't it? * The main motivation is performance (training speed mostly). So some absolute number of e.g. training time would be nice to have in the comparisons. He e.g. mentions that the Adaptive Transformer can also be trained on a single GPU within hours, and in the comparison, the Adaptive Transformer gets much better BPC (enwik8), and uses even slightly less parameters. So, isn't the Adaptive Transformer thus better in every aspect (speed and BPC)? Or how does it compare in speed? As far as I remember, also the Sparse Transformer is more efficient (as it has sparsity), so again the speed comparison would be interesting here. Or is the argumentation for inference speed? But then the inference speed should be compared, or not?
支持赞赏论文的可复现性和写作风格，批评靠堆算力刷榜的风气
说实话，我希望所有研究论文都这样写：易懂、有趣，给出有意义的结果且可在单 GPU 上复现。我知道不是所有深度学习论文都能在合理时间内用单 GPU 复现，但这应该更常见。只是靠增加参数和堆算力刷出新基准显得很偷懒，我更想看到'我们用新设计刷出新基准，旧方法有这些问题'这样的论文。总之，值得一读。
citilife查看英文原文
Honestly, I wish all research papers were written this way. Easy to understand, kept me entertained, and presented meaningful results with a way to reproduce (on a single GPU). I grant all research papers on deep learning can't be reproducible with a single GPU in a reasonable time, but it should happen more often IMO. It seems lazy to just toss out a paper saying "we hit new benchmarks, by increasing the parameters and throwing more compute". I'd like to see "we hit new benchmarks with a new design, the old ones had this issue, etc. Anyway, great read, recommend. Also, happy for the author haha "The author has also moved to a one bedroom apartment in San Francisco, removing themselves from proximity to the alley of questionable odors and unsavory noises."
补充补充作者背景，说明非正式风格不代表工作不严肃
给不熟悉作者的人介绍一下：他之前在 metamind / Salesforce Research 做 NLP，发表过很多成功的 NLP 论文。他这次选择用非正式风格写这篇论文（类似 YOLOv3），但这项工作本身仍应被认真对待。
czr查看英文原文
for those who aren't familiar with the author, he previously worked at metamind / salesforce research doing nlp and has published many successful nlp papers [0]. he opted to write an informal paper for this project (similar to yolov3 [1]), but the work itself should still be taken seriously. [0] https://scholar.google.com/citations?user=AolIi4QAAAAJ [1] https://pjreddie.com/media/files/papers/YOLOv3.pdf
亲历亲历者称赞作者此前 QRNN 工作的实用价值
你就是会喜欢 Stephen Merity。他做的 QRNN 工作在我本科做语言模型毕业论文时帮我省了不少时间和钱。这个 SHA-RNN 似乎源自与 QRNN 相同的思路。
1maginary查看英文原文
You just have to love Stephen Merity. His work on QRNN's saved me quite a bit of time and money when I was doing my undergrad dissertation on language models. This SHA-RNN seems to have surfaced from a similar line of thinking that spawned the QRNN.
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
- 评论区提到的论文
历次讨论
- 2019Hacker News 讨论帖212 分 · 37 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选