AI 与机器学习2025arxiv.org
SWE-bench 数据集的一些关键问题
Some critical issues with the SWE-bench dataset
进阶 · 约 30 分钟读完前置知识：软件缺陷修复与代码补丁基础、SWE-bench 等代码基准的评测流程、单元测试/测试用例概念
导读
这篇论文指出 SWE-bench 数据集存在严重质量问题：部分任务的问题描述不完整、测试用例不足，导致 AI 生成的补丁即使正确也可能被误判为失败。作者过滤掉这些有问题的任务后，SWE-Agent+GPT-4 的解决率从 12.47% 骤降至 3.97%，说明现有基准可能高估了模型的代码能力。
HN 关注点：HN 读者长期关注 LLM 编程能力是否被基准夸大，这篇论文提供了数据集污染导致分数虚高的具体证据，引发对当前所有编程基准可信度的讨论。
HN 评论精选
这几条评论里：支持 1 质疑 1 补充 2
质疑论文指出的“AI 补丁错误”案例经核实多数是误判，AI 补丁其实与人类补丁等效
论文中的一些例子似乎有误。对于 django-31056，作者称 AI 补丁“不完整”是因为缺少 try-except 块和运行中事件循环的检查，但看 diff 就知道这明显不对——这些代码在补丁之前就已存在，人类补丁只是缩进了它们才在 diff 中同时显示为增删。在我看来 AI 补丁是正确的，人类补丁更自然但两者应算平手。对于 django-32517，作者称人类和 AI 补丁“产生完全不同的输出”，但实际上它们做的事完全一样：reversed(self.dict) 和 reversed(self.dict.keys()) 等价，因为迭代字典就是迭代键。人类写法更地道，但也更容易让人困惑——连论文作者都被搞糊涂了。这条也应算平手。
comex查看英文原文
Some of the examples in the paper seem to be wrong. For django-31056, they claim the AI-generated patch is "incomplete" because it's "missing critical parts of this logic, such as the try-except block and the check for a running event loop.". But if you look at the diff, that's clearly wrong. The try-except block and running check were already there before the patch. The human patch just indented them, making them appear as both - and +, while the AI patch didn't. To me, the AI patch seems correct. It's slightly less efficient than the human patch when DJANGO_ALLOW_ASYNC_UNSAFE is set, but slightly more efficient when it isn't (which is the common case!). The human patch does feel more natural, but the AI patch is fine. I'd grade it a tie between human and AI. For django-32517, they claim that the human and AI patches "produce entirely different outputs", but actually they do exactly the same thing. The human version has `reversed(self.dict)`, while the AI version has `reversed(self.dict.keys())`. `reversed` treats the object as an iterator, and iterating over a dictionary in Python just gives you the keys, so it doesn't matter whether you call `.keys()` first. The human patch is more idiomatic, but it's also more confusing, as shown by the fact that it confused the authors of this paper. I'd grade it another tie. Edit: I tried to sign up for OpenReview so I could leave a comment about this, but the system wouldn't let me register without completing a form that assumes you have an academic position. Perhaps I should email the authors.
支持过滤问题任务后分数暴跌，印证了当前编程基准普遍失真的直觉
“当我们过滤掉这些有问题的任务后，SWE-Agent+GPT-4 的解决率从 12.47% 降至 3.97%。”这更符合我对这些模型编程能力的直觉。我认为目前没有任何编程基准能准确衡量编程能力。
modeless查看英文原文
> When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%. This matches my intuition about the coding performance of these models a lot better. I don't think any current coding benchmark accurately measures coding performance.
补充各大公司引用的基准普遍被污染，实际编程体验与高分严重不符
我认为大 LLM 公司引用的几乎所有热门基准都被污染了。OpenAI、xAI、Anthropic、Google 的模型在基准上得分都很高，但你真去让它写代码时，表现也就一般。他们声称模型能达到博士级推理水平，可我连基本的计算思维都不敢信任它。
bearjaws查看英文原文
I would argue almost every popular benchmark quoted by the big LLM companies is tainted. OAI, xAI, Antropic, Google all score incredibly well, then you go to try and write code and its just okay. They claim it can do PHD level reasoning, but here I am not trusting it on basic computational thinking.
补充建议建立定期更新、众包交叉验证的编程评测数据集，避免训练数据污染
我们需要的是一种版本化的众包编程 LLM 评测数据集。每个季度由几千名志愿者各自提供 2 个近三个月内、解决难度不低且测试用例充分的 GitHub issue，志愿者之间互相交叉检查，作为回报获得某 AI 服务的一个月免费订阅。数据集以 SWE-UberBench-2025-02 之类的名字发布，这样大家只能在模型训练期之后发布的数据集上评测自己的编程 LLM。
semi-extrinsic查看英文原文
So what we need is something like a versioned crowdsourced coding LLM eval dataset. Every quarter, you have a couple thousand volunteers provide 2 GitHub issues from the past 3 months, which are nontrivial to resolve, and where there exists strong test cases. Each volunteer then cross-checks 2 issues from other volunteers. The volunteers get 1 month free subscription to some AI service in return. This dataset is then published as SWE-UberBench-2025-02 or something. People can then only evaluate their coding LLM on datasets published after their training period.
历次讨论
- 2025Hacker News 讨论帖350 分 · 116 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选