AI 与机器学习2024arxiv.org
Transformer 配上正确的嵌入就能做算术
Transformers Can Do Arithmetic with the Right Embeddings
进阶 · 约 60 分钟读完前置知识：Transformer 基础、位置编码、算术任务的 tokenization
导读
这篇论文研究如何让 Transformer 更好地完成算术运算。作者通过在嵌入中加入位置/数位等强归纳偏置（如 ABLE 编码），使模型在 100 位数加减等任务上达到约 99% 的准确率，并称该方法对排序等相关任务也有帮助。
HN 关注点：HN 读者争论这是否只是针对特定任务的 hack：有人认为用 Transformer 做算术毫无实用价值，也有人认为这种几乎零成本的嵌入改进值得纳入标准 tokenization。
HN 评论精选
这几条评论里：支持 1 质疑 2 补充 1
质疑质疑研究价值：用神经网络做算术是对已有成熟技术的倒退
这项工作的意义何在？100 位算术 99% 的准确率意味着没人会把 Transformer 当 ALU 用。我们早就会造（字面上）精确得多的加法机器——从 1850 年代帕斯卡的 Pascaline 起就懂机械化算术。文中提出的强归纳偏置对乘法勉强有效，对减法和除法根本不行。花大力气把模式硬塞进神经网络，去重新发明一个残缺、有 bug 的算术机器，这有什么意义？
YeGoblynQueenne查看英文原文
What is the point of this work? 99% on 100-digit arithmetic means there's a 0% chance anyone will ever use a Transformer as an ALU or anything of the kind. We already know how to hard-code a (literally) infinitely more accurate addition machine. And not only addition: all four arithmetic operations. The technique proposed in the article -imposing a strong inductive bias for addition- kiind of works for multiplication, but not for subtraction or division (clearly; I can't even find the words in the paper). As a practical way to build a machine to do arithmetic this is out of the question. We've known how to mechanise arithmetic since the 1850's with Blaize Pascal and his Pascaline. What is the point in demonstrating it's possible to reinvent a broken, partial, buggy version of an arithmetic machine if one tries really hard and shoehorns the necessary patterns in a neural net? We've known that for a long time, too (every proof that a neural net can simulate this or that Turing machine if you design the network diagram and set the weights by hand, ever). So what is the point of this? Transformers are supposed to be the "sparks of AGI" and they can almost do arithmetic if we try very hard to shove it down their heads? Who cares?
质疑认为这是针对特定任务的 hack，回避了真正的推理问题
说实话这看起来像个 hack。问题本来不是让 Transformer 做 100 位数加法，而是现有系统无法进行推理（包括数学）。针对特定用例优化不会带我们去往想去的地方——我们想要的是能学会推理的系统。
msoad查看英文原文
It seems like a hack to be honest. Problem at hand is not to make transformers do addition of 100 digit numbers. Problem is the current systems can’t reason about things, math included. Optimizing for a certain use case is not gonna take us where we wanna be. We want to have a system that can learn to reason.
补充建议用让模型写代码执行的方式替代直接算术
既然模型很擅长写短小的计算机程序，而程序又很擅长数学计算，让它们识别出 'x+y' 这类问题，然后回答'写并执行一个小程序来计算 x+y 再给出结果'，岂不是高效得多？
Shrezzing查看英文原文
Since models are very good at writing very short computer programs, and computer programs are very good at mathematical calculations, would it not be considerably more efficient to train them to recognise a "what is x + y" type problem, and respond with the answer to "write and execute a small javascript program to calculate x + y, then share the result"?
支持持支持态度：低成本高收益的嵌入改进值得推广
这里好多怨气。如果只是加约 20 个 token 来编码列位置/十进制位置，就能让带外任务的数学性能提升三倍，那是大事。这是个简单修复、提升巨大，而且不只是花招——模型还能借此在排序、列表等任务上做得更好。这几乎零成本，没理由不成为标准 tokenization 的一部分。我更感兴趣的是：还能找到哪些类似的有用概念注入嵌入空间？
vessenes查看英文原文
Wow, a lot of grumpiness in here. If it's true that adding like 20 or so tokens to encode column location / decimal spot triples math performance in out of band tasks, that's a big deal. It's a simple fix, it improves performance A LOT, and they even indicate it's not just a party trick, in that the LLM can use the information to do better on related tasks like sorting and list making. This is basically free to add, and there's no reason it shouldn't be made part of standard tokenization. I'm more interested in the question of how we can find other useful concepts for data -> embedding space like this; can we incept our tokenization inception so it has more inception?
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2024Hacker News 讨论帖207 分 · 211 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选