AI 与机器学习2024arxiv.org
TokenFormer：用令牌化模型参数重新思考 Transformer 扩展
TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters
进阶 · 约 75 分钟读完前置知识：Transformer 架构、注意力机制、模型缩放
导读
这篇论文提出 TokenFormer，将 Transformer 中的权重矩阵本身也视为 token，用注意力机制（论文称为 Pattention）来动态计算权重：通过查询权重 key/value 来生成权重。这样扩大模型只需增加更多权重 token 并增量训练，从而把模型缩放变成一种可增量进行的操作。
HN 关注点：是因为它重新思考了 Transformer 的缩放方式——把参数 token 化以支持增量扩展，讨论焦点集中在其实际效果、推理开销以及方法是否只是换包装的 MLP。
HN 评论精选
这几条评论里：支持 1 质疑 2 补充 1
支持赞赏核心思路：用注意力即时生成权重，使扩容变成增量添加权重 token
作者用注意力机制分解每个权重矩阵：weight = attention(token_query, weight_keys, weight_values)。也就是说，查询 weight_keys 来获取 weight_values 并混合，即时计算每个权重。扩大模型只需增加更多 weight_keys 和 weight_values 并增量训练。简单、巧妙，而且看起来有效，很漂亮。
cs702查看英文原文
The authors factorize every weight matrix with an attention mechanism: weight = attention(token_query, weight_keys, weight_values). In other words, they query weight_keys to fetch the weight_values, and mix them to compute each weight on the spot. Increasing model size becomes a matter of adding more weight_keys and weight_values, and incrementally training them. Simple, clever, and it seems to work well. Beautiful.
质疑质疑推理开销：动态计算权重可能拖慢推理速度
我想看看普通 Transformer 和这个方法在推理时间计算上的对比。我猜 token/s 会更低，因为在实际注意力计算之前，需要先为每个 token 计算模型权重。
valine查看英文原文
I would like to see a comparison for the inference time compute between a regular transformer and this. I’m assuming token/s is lower since you need to compute the weights of the model for each token prior to the actual attention calculations for the sequence position.
质疑强烈质疑：认为所谓创新只是带 softmax 非线性的两层 MLP，包装成注意力
作为业内从业者，这篇论文很不幸是彻底的 BS。他们的理论创新其实是：把输入 X 变换到不同维度时，不用简单的线性投影 Y = X @ W，而是引入中间维度和非线性：Y = f(X @ W1) @ W2，f 选了 softmax 以 superficially 类似注意力（后来论文承认其实是修改版 softmax 以避免梯度消失）。他们把注意力层的所有投影都换成这种两层结构。问题不在理论层面……
sapphire42查看英文原文
As someone who has worked in this space, this paper is unfortunately total BS. Their claimed theoretical advancement is as follows. If you want to transform an input vector X to another vector Y of different dimension, "normal" people suggest to use a linear projection: create an appropriately sized matrix W and simply multiply it by your input: Given X ∈ d_in and W ∈ d_in × d_out, then Y ∈ d_out = X @ W. In the attention layer, where the input X is converted into queries Q, keys K, and values V, this is the simple strategy employed: Q = X @ W_q, K = X @ W_k, V = X @ W_v, and it has shown itself to be effective. This is too simple for the authors of this paper. They propose another approach. Instead of converting directly to the desired dimension, we will increase computation by creating an intermediate dimension, and introducing a non-linearity between them. Given X ∈ d_in, and W_1 ∈ d_in × d_tmp, and W_2 ∈ d_tmp × d_out, then Y ∈ d_out = f(X @ W_1) @ W_2. Here, f can be any non-linearity. The authors choose softmax; it allows them to claim a superficial resemblance to attention. Later in the paper, they reveal it is not actually softmax, but a modified version to avoid gradient vanishing (softmax is not a very good general-purpose non-linearity). So, they replace all projections in the attention layer with this new strategy. So Q = f(X @ W_q1) @ W_q2. And K = f(X @ W_k1) @ W_k2. And V = f(X @ W_k3). The problem with this is not theoretical: this does increase the model's expressiveness and computational power. It is practical: we are adding parameters where we need them the least, in the attention layer. It is generally understood that LLMs do not need extra parameters in the attention layer. Actually, advancements like Grouped-Query Attention hinge on the idea that you can halve or even fourth the number of parameters in the attention layer without harming performance. The experience of the LLM community so far suggests that the authors' idea of adding even more parameters to the self-attention layer should degrade their models' performance while adding no tangible gain. The authors' numbers say otherwise. But it is hard to trust their numbers. When training a Transformer to compare against they replicate the original GPT-2 proposed in 2019. In doing so they ignore years of architectural improvements, such as rotary positional embeddings, SwiGLU, and RMSNorm that have culminated in Transformer++, the strong recipe which is what Meta's Llama series uses. We've seen this time after time in the various "Transformer killers" that used to be popular about a year ago. A researcher would think up some novel variant of linear attention, furiously test it against a weak GPT-2 baseline, find it blew it out of the water, and declare victory. Somehow, these never caught on, because when tested against a newer baseline these models weren't actually that great. The authors are doing the same thing here. In their tables they also include comparisons to other models. Actually, they exclusively select the EleutherAI suites: GPT-Neo, OPT, and Pythia. These models were not trained with any modern architectural improvements except rotary embedding (which EleutherAI invented), and so predictably TokenFormer crushes them. On the last page of the appendix the authors have included a full table with some more fair comparisons. Their TokenFormer-150M variant achieves a Pile ppl of 10.45 against Mamba-130M's 10.54. In the intermediate weight class, TokenFormer-450M matches Mamba-370M's 8.28 Pile ppl despite having 21% more parameters. And in the largest size, TokenFormer-1.5B loses to Mamba-1.4B, 6.91 to 6.80 ppl. Overall, the architectural tweak proposed in this paper is impractical, and the few fair comparisons they include are unimpressive. TokenFormer is another in a long line of Transformer-killers that have nice graphs of cherry-picked data, and will similarly fade into obscurity.
补充看好其模块化潜力：权重 token 的顺序或可用于剪枝和增量微调
看起来意义重大。如果公开权重集使用相似的通道维度，这可能实现新层次的模块化和兼容性。也许还为微调提供了很好的形式化框架，比如在 Pattention 层中增删 key-value 对。有趣的是，模型缩放似乎是通过在 K/V 矩阵中添加行来动态进行的，这意味着前面行的权重可能比后面行的更重要——这种权重排序在剪枝等方面可能大有可为。
davesque查看英文原文
Seems like a big deal. I feel like this could enable a new level of modularity and compatibility between publicly available weight sets, assuming they use similar channel dimensions. Maybe it also provides a nice formalism for thinking about fine tuning, where you could adopt certain heuristics for adding/removing key-value pairs from the Pattention layers. One interesting thing to note: sounds like model scaling happens on the fly by adding key-value pairs as rows in the K and V matrices on the Pattention layer. That suggests that weights represented by tokens in the first rows may be more important than weights in later rows. There may be a lot you could do with that ordering of weights in terms of pruning and such.
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2024Hacker News 讨论帖174 分 · 33 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选