AI 与机器学习2017arxiv.org编辑精选
理解深度学习需要重新思考泛化
Understanding deep learning requires rethinking generalization
进阶 · 约 60 分钟读完前置知识：机器学习基础、泛化与模型容量理论、神经网络训练流程
导读
这篇论文研究深度神经网络的泛化能力为何难以用传统模型容量理论解释。核心实验显示，网络可以完美拟合随机标签甚至随机像素，说明其泛化表现不能仅归因于容量控制，需要重新思考“网络究竟学到了什么”。
HN 关注点：挑战了“深度网络为何能泛化”的常识，并引发网络是记忆训练数据还是学习可迁移特征的讨论。
后来怎么样
它成为深度学习泛化讨论的标志性文献，把“网络能记住随机标签”确立为必须解释的现象，并直接催生了记忆容量、双下降、隐式正则化等一批后续研究。
HN 评论精选
这几条评论里：质疑 1 补充 2
质疑认为结果不意外，网络可能存储训练数据并通过复杂变换匹配，而非真正捕捉底层模式。
很高兴看到有人测试神经网络的极限，而不是只在人工基准上挤几个百分点。不过这个结果真那么意外吗？尤其考虑到2015年愚弄DNN的论文和Deep Dream可视化。除非相信网络会“作画”，Deep Dream表明网络捕获并存储训练数据片段，足够聪明就能取回。另一篇论文表明训练好的DNN能把精心构造的噪声以高置信度分类为特定标签，这暗示DNN可能通过某种复杂变换做匹配，通常得到正确答案，但不一定捕捉底层模式。就像靠迹象猜天气，却不懂气压、气流等。
gambler查看英文原文
Good to see someone testing the limits of neural nets, rather just squeezing a few percent of performance on an artificial benchmark. That said, is this result really all that surprising? Especially given the results demonstrated in that paper on fooling DNNs from 2015 and visualization experiments a-la Deep Dream. Unless you believe in networks "painting" stuff, Deep Dream demonstrated that neural networks capture and store certain chunks of their training data and you can get those back out if you're clever enough. That other paper[1] demonstrated that a trained DNN can classify noise as a particular label with very high confidence, as long as you construct that noise carefully enough. This hints at the fact that DNNs may do matching by applying some complex transformation that usually results in the correct answer, but does not necessarily capture the underlying patterns. (Kind of like guessing about the weather by telltale signs, without knowing anything air pressure, currents and so on.) [1] - http://www.evolvingai.org/fooling
补充用图1(a)解释卷积架构偏好几何特征，随机标签需记忆容量，打乱像素靠颜色直方图。
我们阅读小组上周讨论过这篇论文。我认为理解这里发生了什么的关键是图1(a)。真实标签学习最快，随机标签最慢，打乱像素第二快。我认为原因在于：给定由结构化图像组成的训练数据，卷积架构强烈偏好学习反映几何特征的滤波器，而不是能记忆数据的随机滤波器。因此真实标签学习最快，因为几何特征对应学习目标；但记忆随机标签时，几何特征的容量低于随机滤波器。另一方面，打乱像素学得相当快，因为卷积架构容易捕捉颜色直方图并据此学习。
AlexCoventry查看英文原文
We discussed this paper in our reading group last week[0]. I think the key to understanding what's going on here is figure 1(a). The fastest learning happens with true labels, and the slowest with random labels. Shuffled pixels is the second fastest. I believe the reason this is happening is that given training data composed of structured images, the convolutional architecture heavily favors learning filters which reflect geometric features, as opposed to random filters which can memorize the data. This results in fastest learning with the true labels because the geometric features correspond to the learning target, but for memorizing random labels, geometric features have lower capacity than random filters. On the other hand, it learns shuffled pixels pretty fast because the convolutional architecture makes it easy to capture a color histogram and learn off that. [0] This week we discussed the Alpha Go paper. URL for that, although we don't generally advertise our meetings unless we think there's going to be broad interest: https://www.meetup.com/Cambridge-Artificial-Intelligence-Meetup/events/237183581/
补充认为核心谜团是网络为何偏好可理解模型，猜测存在模型排序。
仅从摘要的半懂解读——现代图像识别网络能学习随机分配给随机图像集的标签，这意味着当给定非随机图像时，它们为何学习有可理解含义的标签仍然神秘，而不是通过某种无意义模型只记忆训练集。我猜解决方案必须涉及对可能模型的排序，其中对设计良好的网络，可理解模型优先于不可理解模型。先存着以后读。
maxander查看英文原文
My halfway informed interpretation, just from the abstract- it turns out that modern image-recognition networks are capable of learning labels randomly assigned to sets of random images, which means that it's still mysterious why they learn labels with intelligible meaning when given non-random images (rather than just memorizing the training set via some nonsense model.) I'd guess the resolution would have to involve an ordering over possible models, where (for well-designed networks) intelligible models are preferred over unintelligble ones. Filing this away to read later.
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2017Hacker News 讨论帖131 分 · 18 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选