AI 与机器学习2024arxiv.org
RNN 就是我们所需要的一切吗？
Were RNNs all we needed?
进阶 · 约 60 分钟读完前置知识：RNN/LSTM 基础、Transformer 基础、并行训练与序列建模概念
导读
这篇论文重新审视 RNN 是否足以替代 Transformer：作者提出 minLSTM 和 minGRU 等简化循环单元，去掉门控对前一隐藏状态的依赖，使训练阶段可以像 Transformer 一样并行化，同时保留推理时的循环结构。论文报告这些模型在若干序列建模任务上以更少参数、更少训练步数达到与 Transformer 相当的性能。
HN 关注点：触及 Transformer 是否不可替代、架构选择在深度学习中到底有多重要，以及循环模型能否兼顾并行训练和长上下文。
HN 评论精选
这几条评论里：质疑 2 补充 2
补充转述 Chollet 观点：在数据规模足够大时，架构差异会收敛，关键在数据集和模型表达能力。
这个话题在这么多地方讨论既是诅咒也是祝福。我在 Twitter/X 上看到一条有趣的评论：有趣的工作，复兴 RNN。总体而言，最近来自不同方向、大致能匹配 Transformer 的许多架构证明，在曲线拟合范式（即深度学习）中，架构并不是根本重要的。曲线拟合就是把数据集嵌入到一条曲线上。关键因素是数据集，而不是那些限制曲线形状的硬编码花哨组件。只要你的曲线足够有表达力，在大数据条件下所有架构都会收敛到相同性能。
xnx查看英文原文
It's curse and a blessing that discussion of topics happens in so many different places. I found this comment on Twitter/X interesting: https://x.com/fchollet/status/1841902521717293273 "Interesting work on reviving RNNs. https://arxiv.org/abs/2410.01201 -- in general the fact that there are many recent architectures coming from different directions that roughly match Transformers is proof that architectures aren't fundamentally important in the curve-fitting paradigm (aka deep learning) Curve-fitting is about embedding a dataset on a curve. The critical factor is the dataset, not the specific hard-coded bells and whistles that constrain the curve's shape. As long as your curve is sufficiently expressive all architectures will converge to the same performance in the large-data regime."
质疑质疑去掉隐藏状态后模型是否还真的是 RNN，还是退化成逐 token 的 MLP。
我还没细读论文，但也许有人能回答：如果像他们说的那样把 RNN 的隐藏状态去掉，那还剩下什么？一个只根据单个 token 做预测的 MLP 吗？
tehsauce查看英文原文
I haven’t gone through the paper in detail yet but maybe someone can answer. If you remove the hidden state from an rnn as they say they’ve done, what’s left? An mlp predicting from a single token?
补充用 IIR/FIR 滤波器类比说明循环架构参数效率更高，但前馈更易实现，并认为长期需要根本性循环或神经形态架构。
我想类比数字信号处理：把循环式架构看作 IIR 滤波器，把纯前馈架构看作 FIR 滤波器，会发现很多相似之处。最明显的是，IIR 滤波器通常用少得多的元件就能产生与等效 FIR 滤波器相同的响应。当然，FIR 滤波器在实际中往往更容易实现、控制、测量（定点算术硬件对应能在 GPU 上运行的 ML 架构）。我不认为没有某种根本上的循环架构，我们能到达 AI 那可怕的指数级阶段。LSTM 之类在这个 DSP 类比里像中间 hack，可以看作带动态系数的 FIR。长期看，我认为神经形态方法在效率上是最佳赌注。
bob1029查看英文原文
> Transformers required ~2.5x more training steps to achieve comparable performance, overfitting eventually. > RNNs are particularly suitable for sequence modelling settings such as those involving time series, natural language processing, and other sequential tasks where context from previous steps informs the current prediction. I would like to draw an analogy to digital signal processing. If you think of the recurrent-style architectures as IIR filters and feedforward-only architectures as FIR filters, you will likely find many parallels. The most obvious to me being that IIR filters typically require far fewer elements to produce the same response as an equivalent FIR filter. Granted, the FIR filter is often easier to implement/control/measure in practical terms (fixed-point arithmetic hardware == ML architectures that can run on GPUs). I don't think we get to the exponential scary part of AI without some fundamentally recurrent architecture. I think things like LSTM are kind of an in-between hack in this DSP analogy - You could look at it as FIR with dynamic coefficients. Neuromorphic approaches seem like the best long term bet to me in terms of efficiency.
质疑质疑 RNN 的固定大小隐藏状态限制了长上下文信息保留，因此不能普遍替代 Transformer。
我的感觉是答案是“不”，意思是这些 RNN 无法在 LLM 中普遍取代 Transformer，尽管在某些情况下可能够好、在另一些情况下甚至胜出。原因如下：LLM 用户可能给模型一段长文本然后说“请把它翻译成德语”。Transformer 可以回看它的全部历史。但 RNN 能做什么？虽然它的上下文长度无限，但模型对上下文保留的信息量受限于任一时刻隐藏状态里的内容。相关：https://arxiv.org/abs/2402.01032
trott查看英文原文
My feeling is that the answer is "no", in the sense that these RNNs wouldn't be able to universally replace Transformers in LLMs, even though they might be good enough in some cases and beat them in others. Here's why. A user of an LLM might give the model some long text and then say "Translate this into German please". A Transformer can look back at its whole history. But what is an RNN to do? While the length of its context is unlimited, the amount of information the model retains about it is bounded by whatever is in its hidden state at any given time. Relevant: https://arxiv.org/abs/2402.01032
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2024Hacker News 讨论帖520 分 · 260 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选