AI 与机器学习2020arxiv.org
梯度下降学习得到的每个模型都近似为核机器
Every Model Learned by Gradient Descent Is Approximately a Kernel Machine
进阶 · 约 40 分钟读完前置知识：核方法与 SVM 基础、梯度下降/SGD、多元微积分与路径积分
导读
这篇论文提出，任何通过梯度下降（包括随机梯度下降）训练得到的模型，无论网络架构如何，都近似等价于使用一种“路径核”（path kernel）的核机器。作者定义的路径核衡量两个数据点在训练过程中模型梯度变化的相似程度。它更像是一个理论上的形式等价，而非可直接带来计算加速的实用结论。
HN 关注点：深度学习与 SVM/核方法之间是否真有深层联系，以及这种等价能否解释泛化或带来计算优势；讨论焦点很快转向其理论意义与计算可行性。
HN 评论精选
这几条评论里：支持 1 质疑 2 补充 1
质疑质疑：通用逼近器理论早已存在，关键在训练算法，该结论并未回答这一核心问题。
我不太理解这类结论，希望有人能科普。已知有很多通用逼近器：深度网络、SVM、三次样条（用了近百年）。问题从来不是找不到足够强的逼近器，而是如何训练它。据我理解，深度学习的重要进步是我们终于能训练某种通用逼近器，使其为过去难以解决的分类问题找到很好的分隔面。理论上存在其他通用逼近器能近似复现同样的分隔面，这不应令人惊讶吧？任何“通用”逼近器都该足够强。问题一直是找对权重，不是找对架构。我漏了什么吗？
GlenTheMachine查看英文原文
I'm confused by findings like this one, and I'm hoping someone here can educated me. There are many known universal approximations. Deep networks are one. SVMs are one. Heck, cubic splines are one, and they've been in use for nearly a hundred years IIRC. The problem has never been one of finding a sufficiently powerful approximator. It has been training that approximator. My understanding of the significant advancement made by deep learning is that we finally figured out how to train a specific kind of universal approximator in a way such that it finds very good separation surfaces for what used to be impossible-to-solve classification problems. But it should be no surprise to anyone that there exist, in theory, other universal approximations that approximately reproduce the same separation surfaces, should it? I'd expect any universal approximator to be powerful enough to reproduce the separation surfaces, hence the meaning of the word "universal". The problem was always finding the right weights, not finding the right approximator architecture. Am I missing something?
支持专业解读：路径核不是普通RBF核，价值在于揭示SGD模型与SVM的深层理论联系。
作者说：用SGD学到的每个模型，不论架构，都近似等价于带特定类型核的核机器（如SVM），Domingos称之为“路径核”。按论文定义，路径核衡量任意两个数据点上模型在SGD训练期间变化（具体是梯度变化）的相似程度。这不是常见的高斯/RBF核。我们早就知道SVM是通用逼近器；这项工作的意义在于发现SGD训练出的任意模型与理解较充分的SVM之间新的、出人意料的深层联系。
cs702查看英文原文
"Here we show that every model learned by this method [SGD], regardless of architecture, is approximately equivalent to a kernel machine [i.e., a support vector machine or SVM] with a particular type of kernel" -- a type of kernel which Domingos, the author, calls a "path kernel." As defined in the paper, a "path kernel" measures, for any two data points, how similarly a model varies (specifically, how similarly the model's gradients change) at those two data points during training via SGD. This isn't exactly your usual, plain-vanilla, radial-basis type of kernel. We've known for a long time that SVMs are universal approximators, i.e., in theory they can approximate any target function. The importance of this work is that it has found a new, surprising, deep connection between any model trained via SGD and SVMs, which are well understood :-)
质疑质疑实用性：只是形式等价，路径核推理成本近似等于重新训练，难有计算加速。
对那些说“哇可以回到SVM、大提速”的人：这太早了。这里只是形式等价，计算上并不很有用。数学基本是本科一年级微积分，第2节写得清楚。作者本质上证明：任何用“无限小步长全批量梯度下降”（即沿梯度流）训练的模型，都可写成核化形式 y(x)=Σ a_i K(x,x_i)+b。人们对SVM的直觉是 a_i 恒定、稀疏，K 计算便宜（核技巧）。这里这些性质都不成立，所以计算上不激动人心。诀窍是 a_i 和 K 都涉及沿梯度流的路径积分，因此单次推理大约和从头训练模型一样贵（如果我理解正确）。
ajtulloch查看英文原文
For everyone saying "oh wow, we can go back to SVMs now, huge speedups, etc" - that's more than a little premature. This is purely a formal equivalence, but not very useful computationally. The math here is pretty much first-year undergraduate level calculus, and it's worth going through section 2 since it's quite clearly written (thanks Prof Domingos). Essentially, what the author does is show that any model trained with "infinitely-small step size full-batch gradient descent" (i.e. a model following a gradient flow) can be written in a "kernelized" form y(x) = \sum_{(x_i, y_i) \in L} a_i K(x, x_i) + b. The intuition most people have for SVMs is that the constants a_i are, well, constant, that the a_i are sparse, and that the kernel function K(x, x_i) is cheap to compute (partly why it's called the 'kernel trick'). However, none of those properties apply here, which is why this isn't particularly exciting computationally. The "trick" is that both a_i and K(x, x_i) involve path integrals along the gradient flow for the input x, and so doing a single inference is approximately as expensive as training the model from scratch (if I understand this correctly).
补充补充资料：有研究显示CNN在ImageNet上的优势可能主要来自微调，而非根本不同的决策策略。
另见 Brendel 和 Bethge 的“Approximating CNNs with bag-of-local-features models works surprisingly well on ImageNet”（本文引用），该文称：过去几年DNN相对旧bag-of-feature分类器的提升，主要来自更好的微调，而非质上不同的决策策略。
sarosh查看英文原文
See also W. Brendel and M. Bethge. "Approximating CNNs with bag-of-local-features models works surprisingly well on ImageNet." https://arxiv.org/abs/1904.00760 (which is referenced in this paper) and explains "[t]his suggests that the improvements of DNNs over previous bag-of-feature classifiers in the last few years is mostly achieved by better fine-tuning rather than by qualitatively different decision strategies."
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2020Hacker News 讨论帖406 分 · 107 评
- 2024Hacker News 讨论帖176 分 · 136 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选