AI 与机器学习2019arxiv.org
对近期神经推荐方法的令人担忧的分析
A Worrying Analysis of Recent Neural Recommendation Approaches
进阶 · 约 60 分钟读完前置知识：推荐系统基础、协同过滤、实验评估与基线
导读
这篇论文对近年发表在顶级会议上的 18 个神经网络推荐算法做复现分析：由于代码或数据缺失，仅 7 个能以合理成本复现；其中 6 个的所谓提升可被最近邻、图方法等简单启发式基线追平或超越，剩下 1 个（Mult-VAE）也未稳定胜过调优后的非神经线性排序方法。作者据此质疑该领域常见实验设置与发表实践。
HN 关注点：机器学习推荐系统乃至整个 ML 领域的可复现性危机：弱基线、代码不公开和评审激励如何制造“进展”。
后来怎么样
该文成为推荐系统领域可复现性讨论的标志性工作，推动后续会议和期刊更重视代码/数据公开与可复现评审，并促使研究者采用更强基线。
HN 评论精选
这几条评论里：支持 2 补充 2
支持复现失败主要暴露代码数据缺失与弱基线包装，应提高可复现标准并改革评审。
总结一下：他们尝试复现 18 篇近期论文，但因缺少可用代码或数据，只能尝试复现其中 7 篇；在这 7 篇里，有 6 篇的“改进”用更简单的非机器学习方法（如最近邻）也能取得。第 7 篇（Mult-VAE）则可以通过更好地调参该文使用的一个基线，几乎追平其改进幅度。这并不意味着已发表结果都是垃圾，但首先别错过主要结论：不到一半的论文有足够代码和数据可供复现；如果期刊和审稿人要求，这个数字肯定能改善。其次，我猜这意味着作者会用过度包装来提高录用概率：用弱基线并宣称提升 50%，比用现实基线宣称 0–5% 更容易发表，而他们可能确实没猜错。有用的改变是调整论文可发表性的评判标准，审稿人应反对弱基线，同时更接受……
nkurz查看英文原文
Summarizing, they tried replicating the results of 18 recent papers, and determined that because of lack of available code or data, they were only able to attempt replications of 7 of them. Of these, they found that the "improvements" in 6 of them could also be achieved with a simpler non-machine learning approach (such as nearest neighbor). For the 7th (Mult-VAE), they were able to almost match the improvement with better tuning of one of the baselines used in that paper. Does this mean that the published results are garbage? No! But what does it mean? First, don't miss the main claim: less than half the published papers have sufficient code and data available for replication. While one might argue exactly what the standards for availability should be, if journals and reviewers were to demand it, this number can definitely be improved. Greater availability of data and code makes for better science. Second, my guess is that it means that paper authors are using excess "puffery" in an attempt to improve the chances that their papers are accepted for publication. They've decided that using a weak baseline and claiming a 50% improvement is more likely to result in publication than using a realistic baseline and claiming a 0-5% improvement. Unfortunately, they are probably right. The useful change (in my opinion) would be to change the standards by which papers are judged to be publishable. Reviewers should push back against weak baselines, but be more accepting of results that make modest claims. You can still have a useful paper even if the degree of improvement is small --- or even nonexistent. Maybe it will inspire someone else, maybe it's useful in an ensemble, or maybe it applies to a case where the baseline wouldn't. It's the puffery that's the problem, not the publication. Creating a system that rewards authors who more honestly evaluate their work would be an overall win.
补充类似问题跨领域存在，复现成本让深度学习研究向少数大实验室集中。
与此相关的是另一篇 arXiv 论文，它发现 BERT 在 Argument Reasoning Comprehension Task 上 77% 的峰值性能只比未训练人类基线低三个点，但这一结果完全可由数据集中的虚假统计线索解释；多种模型都利用这些线索，作者据此构造对抗数据集，所有模型在该数据集上只有随机准确率。计算机视觉领域也有类似怀疑：SOTA 模型过拟合 ImageNet 等数据集；问题在于普通实验室没有数十万美元复现极昂贵、高度调参的模型。深度学习复现与推进正变得除少数资金充裕的工业实验室外几乎不可及。这不全是坏事，只是不能把一切都当福音。
codingslave查看英文原文
Similar to this post is this: https://arxiv.org/abs/1907.07355 "Probing Neural Network Comprehension of Natural Language Arguments Timothy Niven, Hung-Yu Kao (Submitted on 17 Jul 2019) We are surprised to find that BERT's peak performance of 77% on the Argument Reasoning Comprehension Task reaches just three points below the average untrained human baseline. However, we show that this result is entirely accounted for by exploitation of spurious statistical cues in the dataset. We analyze the nature of these cues and demonstrate that a range of models all exploit them. This analysis informs the construction of an adversarial dataset on which all models achieve random accuracy. Our adversarial dataset provides a more robust assessment of argument comprehension and should be adopted as the standard in future work." In the field of computer vision, there are similar suspicions that state of the art computer vision models are overfit to ImageNet datasets and the like. The issue being that even average research labs do not have hundreds of thousands of dollars to reproduce extremely expensive and highly tuned models. Reproducing and advancing the field of deep learning is quickly becoming inaccessible to almost everyone except for a few of the highest funded industrial labs (Google, FB, OpenAI, Microsoft Research among a few others). This is not all negative, it's just that not everything can be taken as gospel.
补充弱基线论文近似负面结果，但表述偏差使领域难以区分真实进展，简单模型可能已足够。
这让人不安地想起心理学和社会科学中的复现危机。我的要点：1）一半论文在技术层面无法复现——请公开代码和数据！2）多数论文使用“弱基线”以显示某种改进并发表。我很矛盾：若要求每篇都打败 SOTA，整个学科一年可能只发一篇。某种角度看这些论文像负面结果——我们试了但没用——并不坏，但带偏差的表述让好坏难以区分。3）不一定还能从这块石头里榨出更多商业价值；有时简单算法就能找到数据中所有有用信息，不是所有东西都受益于更复杂表示，有时调参或更多数据也救不了，只能接受简单模型捕捉了全部信号，其余是噪声。
olooney查看英文原文
Eerily reminiscent of the replication crisis in psychology and the social sciences. My key takeaways: 1) Half the papers couldn't be reproduced on a technical level. Publish your code and your data, people! 2) Most of these papers uses "weak baselines" so they can show some kind of improvement and get their paper published. I'm conflicted about this because if we require every paper to beat state-of-the-art, we'd (collectively, as the entire discipline) be lucky to publish one paper a year. From one point of view, these papers actually represent a form of publishing negative results - we tried this and it didn't work - which isn't a bad thing. But the biased way its presented makes it harder to separate the wheat from the chaff. 3) It's not obvious that we're going to squeeze any more business value from this particular stone. Sometimes all the useful information in a dataset can be found with a fairly simple algorithm. Not everything benefits from a more complex representation, and sometimes you can't fix that with regularization or more data. Sometimes you just have to use the simple model and accept that it captures all the signal that's available and everything else is noise.
支持评审规则奖励弱基线比较，抑制诚实但暂时不占优的新方法。
有些领域（尤其推荐系统）里，如果你报告强基线而你的方法没有超过它，你发不了论文；但如果你用弱基线且方法超过它，就可能发表。这对我说不通，因为 promising new approaches 本来就可能暂时超不过当前 SOTA。这种同行评审偏差激励了坏研究实践。
anjc查看英文原文
There are fields (recommender systems in particular in my opinion) where you will not get published if you report a strong baseline and your approach does not outperform it, but you may get published if you use weak baselines and your approach outperforms it. This doesn't make sense to me as surely it's possible for promising new approaches to not yet outperform current state of the art methods. These peer review aberrations incentivise these bad research practices I think.
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2019Hacker News 讨论帖245 分 · 82 评
- 2019Hacker News 讨论帖86 分 · 17 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选