AI 与机器学习2024openreview.net编辑精选
Vision Transformers 需要寄存器
Vision Transformers Need Registers
进阶 · 约 60 分钟读完前置知识：Vision Transformer 基础、注意力机制、自监督预训练（DINO）
导读
这篇论文研究 Vision Transformer 中的高范数异常 token 现象：这些 token 会出现在特征图中并损害下游任务表现。作者提出在输入序列末尾添加若干可学习的'register' token，让模型把全局信息存储在这些额外 token 中，从而消除伪影、获得更平滑的特征图，并在密集预测和无监督目标发现等任务上显著提升性能，推理开销仅增加约 2%。
HN 关注点：这一近乎'免费午餐'的简单改进，以及它与语言模型中 'attention sink'、填充 token 等现象的关联。
HN 评论精选
这几条评论里：支持 1 补充 2
支持解释 register token 的机制与收益，认为这是接近免费的性能提升
根据论文，'registers' 是训练时附加到 ViT 输入序列末尾的额外可学习 token，类似 [CLS] token，推理时被丢弃。它们为模型在前向传播中存储、处理和检索全局信息提供了场所，避免 patch token 被挪作此用。加入 register token 后，特征图中的伪影和高范数'离群'token 消失，密集预测性能提升，无监督目标发现也更好。推理成本仅增加 2%，几乎是免费午餐。
ttul查看英文原文
According to the paper, the "registers" are additional learnable tokens that are appended to the input sequence of a Vision Transformer model during training. They are added after the patch embedding layer, with a learnable value, similar to the [CLS] token and then at the end of the Vision Transformer, the register tokens are discarded, and only the [CLS] token and patch tokens are used as image representations. The register tokens provide a place for the model to store, process and retrieve global information during the forward pass, without repurposing patch tokens for this role. Adding register tokens removes the artifacts and high-norm "outlier" tokens that otherwise appear in the feature maps of trained Vision Transformer models. Using register tokens leads to smoother feature maps, improved performance on dense prediction tasks, and enables better unsupervised object discovery compared to the same models trained without the additional register tokens. This is a neat result. For just a 2% increase in inference cost, you can significantly improve ViT model performance. Close to a free lunch.
补充联想到语言模型中的 attention sink 现象，提出类比疑问
这是视觉模型版的 'attention sinks' 吗？
superkuh查看英文原文
"Attention sinks" for vision models?
补充补充相关论文：额外 token 可提供与内容无关的计算收益
相关研究？《Let's Think Dot by Dot: Hidden Computation in Transformer Language Models》：transformer 可以用无意义的填充 token（如'......'）代替思维链来解决原本无法解决的算法任务，但学习使用填充 token 较困难，需要特定的密集监督。额外 token 能独立于其内容提供计算收益，这引发了对大模型进行不可审计的隐藏计算的担忧。
richdougherty查看英文原文
Related? "Let's Think Dot by Dot: Hidden Computation in Transformer Language Models" https://arxiv.org/abs/2404.15758 > Chain-of-thought responses from language models improve performance across most benchmarks. However, it remains unclear to what extent these performance gains can be attributed to human-like task decomposition or simply the greater computation that additional tokens allow. We show that transformers can use meaningless filler tokens (e.g., '......') in place of a chain of thought to solve two hard algorithmic tasks they could not solve when responding without intermediate tokens. However, we find empirically that learning to use filler tokens is difficult and requires specific, dense supervision to converge. We also provide a theoretical characterization of the class of problems where filler tokens are useful in terms of the quantifier depth of a first-order formula. For problems satisfying this characterization, chain-of-thought tokens need not provide information about the intermediate computational steps involved in multi-token computations. In summary, our results show that additional tokens can provide computational benefits independent of token choice. The fact that intermediate tokens can act as filler tokens raises concerns about large language models engaging in unauditable, hidden computations that are increasingly detached from the observed chain-of-thought tokens. > In this work, we demonstrate that transformers trained on the next-token prediction objective can achieve improved performance on certain tasks when given filler tokens, achieving perfect accuracy whereas the no-filler, immediate-answer setting achieves only low accuracy. -- I wonder if we could get benefits from adding special computation/register tokens to text LLMs? More discussion: - https://news.ycombinator.com/item?id=40182695 - https://www.reddit.com/r/LocalLLaMA/comments/1cf2w5a/transformers_can_use_meaningless_filler_tokens_eg/
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2024Hacker News 讨论帖171 分 · 23 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选