AI 与机器学习2025openreview.net
Putnam 问题轻微变化时 O1-preview 准确率下降 30%
30% drop in O1-preview accuracy when Putnam problems are slightly variated
入门 · 约 30 分钟读完前置知识：大语言模型基础、基准测试与过拟合概念
导读
这篇论文（OpenReview 投稿）考察 OpenAI 的 O1-preview 在普特南数学竞赛（Putnam）题目上的稳健性：当题目被轻微改写（如换数字、改措辞）时，准确率下降约 30%。作者据此质疑模型在该基准上的高分在多大程度上反映了真实推理能力，而非对原题的记忆或过拟合。
HN 关注点：基准污染与「推理还是背诵」之争：模型高分是否只是训练集里见过原题。
HN 评论精选
这几条评论里：质疑 1 补充 2 亲历 1
亲历亲测改写后的「羽毛vs砖头」题，模型给出自相矛盾的答案，说明其表现依赖训练集记忆，需用分布外测试评估真实能力。
我记得当初这类模型刚出来时，大家很兴奋它终于答对了「10 磅羽毛和 10 磅砖头哪个更重」。可它当然答得对——这题就在训练集里。你把名词换一换，或者改数字让其中一个真的更重，表现就乱七八糟。我刚去 chatgpt.com 输入「9.99 磅钢锭和 10.01 磅蓬松棉花哪个更重？」，第一次的回答（没有反复试）是：9.99 磅钢锭比 10.01 磅棉花略重……因为钢更致密，钢袋 9.99 磅而棉花 10.01 磅，所以蓬松的棉花其实稍微重一点。既对又错，只能判为失败。要分析真实能力，必须确保测试在训练集之外；而你 5 秒内想到的东西之所以跳出来，恰恰因为你自己见得很多、或很容易想到，因此别人也很容易想到……
jerf查看英文原文
I remember when this stuff was all coming out and people were finally excited about ChatGPT getting the problem with "which is heavier, a 10 pound bag of feathers or a 10 pound bag of bricks?" problem correct. But of course it got it correct. It was in the training set. Vary the problem slightly by just changing the nouns, or changing the numbers so that one in fact was heavier than the other, and performance went all over the map. I just went to chatgpt.com and put into the chat box "Which is heavier, a 9.99-pound back of steel ingots or a 10.01 bag of fluffy cotton?", and the very first answer I got (that is, I didn't go fishing here) was The 9.99-pound bag of steel ingots is heavier than the 10.01-pound bag of fluffy cotton by a small margin. Although the cotton may appear larger due to its fluffy nature, the steel ingots are denser and the weight of the steel bag is 9.99 pounds compared to the 10.01 pounds of cotton. So, the fluffy cotton weighs just a tiny bit more than the steel ingots. Which, despite getting it both right and wrong, must still be graded as a "fail". If you want to analyze these thing for their true capability, you need to make sure you're out of the training set... and most of the things that leap to your mind in 5 seconds are leaping to your mind precisely because they are either something you've seen quite often or something that you can easily think of and therefore many other people have easily thought of them as well. Get off the beaten path a bit and the math gets much less impressive.
补充提出用 1905 年前语料训练模型再让它推导 E=mc² 的思想实验，以判定模式识别是否等同智能。
我很想看到一个实验（现实中不太可行）：只用 1905 年之前的所有数字化资料（期刊、信件、书籍、广播、讲义等）训练一个模型，然后让它给出质能等价公式。一个确定的答案就能彻底了结「模式识别算不算智能」的争论 ;)
wim查看英文原文
One experiment I would love to see, although not really feasible in practice, is to train a model on all digitized data from before the year 1905 (journals, letters, books, broadcasts, lectures, the works), and then ask it for a formula for mass-energy equivalence. A certain answer would definitely settle the debate on whether pattern recognition is a form of intelligence ;)
质疑质疑模型高分源于针对基准的硬编码，且用普特南题考聊天机器人本身就不合适。
模型如今只是针对各种随机基准被硬编码，这难道不是公开的秘密吗？人们居然拿普特南题去问聊天机器人，也挺奇怪的 :/
yifanl查看英文原文
Is it just an open secret that the models are currently just being hardcoded for random benchmarks? Seems weird that people would be asking Putnam problems to a chatbot :/
补充用数 a 与 strawberry 的例子说明模型只是被针对性训练修好了已知失败案例，而非真正掌握规则。
一个有意思的例子：句子「How many 'a' in this sentence?」里有 6 个 a（见分享链接）。而常见的「strawberry」版本现在能答对了（有 3 个 r）。显然模型没学到教训，只是被人在失败案例上训练过而已。
frikskit查看英文原文
An interesting example of this is: There are 6 “a”s in the sentence: “How many ‘a’ in this sentence?” https://chatgpt.com/share/677582a9-45fc-8003-8114-edd2e6efa289 Whereas the typical “strawberry” variant is now correct. There are 3 “r”s in the word “strawberry.” Clearly the lesson wasn’t learned, the model was just trained on people highlighting this failure case.
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2025Hacker News 讨论帖560 分 · 526 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选