AI 与机器学习2019arxiv.org编辑精选
任意架构的宽神经网络都是高斯过程
Wide Neural Networks of Any Architecture Are Gaussian Processes
进阶 · 约 75 分钟读完前置知识：中心极限定理、高斯过程、神经网络基础
导读
论文研究宽神经网络在宽度趋于无穷时的行为，主张任意架构的宽网络在初始化时都可用高斯过程描述。其核心直觉可归结为对层间聚合应用中心极限定理，使高阶项在渐近意义下消失。
HN 关注点：把宽网络极限与高斯过程/中心极限定理联系起来，同时讨论“任意架构”是否真能覆盖实际网络、是否只适用于初始化，以及高斯权重初始化假设是否过窄。
后来怎么样
将无限宽网络的 NNGP 框架推广到任意架构，成为无限宽神经网络理论的基础工作之一，并推动了 Neural Tangents 等工具及后续 NNGP/NTK 研究。
HN 评论精选
这几条评论里：质疑 2 补充 1 亲历 1
补充宽网络极限大概率由 CLT 解释，但实际离渐近行为有多近往往不清楚。
没读论文，我打赌归根结底是中心极限定理；而 CLT 本身又归结为三阶及更高阶项在渐近意义下不重要。问题始终是：你从什么时候开始明显接近这个渐近结果？答案常常是我们完全不知道。补充：关于高阶项，我说的是经典 CLT 的证明，即对中心化随机变量的特征函数做泰勒展开并截断到二阶。
throwlaplace查看英文原文
Without reading the paper I bet it comes down to the central limit theorem, which itself comes down to the fact that 3rd order and higher terms don't matter asymptotically. The question is always when do you start appreciably approaching the asymptote (answer: we often have no idea). edit: re higher order terms i'm talking about the proof of the classic clt https://en.wikipedia.org/wiki/Central_limit_theorem#Proof_of_classical_CLT theres a taylor series expansion of the characteristic of the centered rv that's truncated to second order.
亲历Radford Neal 指出论文对其 1994 年结果的表述不准确：他的收敛结果不限于浅层，还研究过无限方差先验下的非高斯稳定过程。
这看起来有意思（目前只扫了一眼）。不过摘要和引言对我 1994 年论文所做工作的描述有点误导。我关于神经网络收敛到高斯过程的结果并不局限于浅层网络。我还讨论了如何通过使用方差无穷的权重先验，让网络先验收敛到非高斯稳定过程。对于“真正复杂的平滑器”不够用的问题，这类非高斯先验可能比高斯过程更有意思。
radford-neal查看英文原文
This looks interesting (have only glanced at it so far). However, the abstract and introduction are a bit misleading regarding what I did in my 1994 thesis (http://www.cs.utoronto.ca/~radford/thesis.abstract.html). My results about convergence of neural networks to Gaussian processes were not confined to shallow networks. Also, I discussed how to get network priors to converge to non-Gaussian stable processes, by using priors for weights that have infinite variance. Such non-Gaussian priors may well be more interesting than Gaussian processes, for problems where a "really sophisticated smoother" is not going to be adequate.
质疑质疑“任意架构”的覆盖范围，并强调讨论对象只是初始化状态、不是训练后的网络。
这听起来重要且有趣，但“宽”不是这里的关键词吗？他们谈到浅层 NN 和深层全连接 NN，但这似乎漏掉了很多。文章提出了一种表达神经网络的独特语言/模型，这很酷，但他们说的是今天见到的大多数 NN 吗？如果是，那仍然很重要。别管了，看 SiempreViernes 的评论：“我找不到任何关于已训练 NN 的提及，这严格来说是初始状态。”
joe_the_user查看英文原文
This sounds important and interesting but isn't wide the key word here? They talk about shallow NNs and deep fully connected NNs but that would seem to leave out a lot. I mean, the article puts forward a distinct language/model to expression neural nets in, which is cool but are they talking about all or most of the NNs you see today? If so, huge but still. Fo-get-a-bout-it, see SiempreViernes' comment: "I couldn't find any mention about a trained NN, this is strictly about the initial state. "(emphasis added) https://news.ycombinator.com/item?id=21653516
质疑质疑高斯权重初始化假设过窄，认为理想目标应推广到一般独立初始化甚至非高斯情形。
好结果事后看往往显得“显然”，所以请把这当作赞美。直觉上（我从没读过这个领域的论文），既然谈的是宽网络，我也预期会用 CLT。对“稠密”层来说，每层聚合可用 CLT 刻画似乎很明显，依此类推；对采样加一些温和的独立性/混合/鞅假设可能就够了。因此“任意架构”论文的目标应是为不同层找到推广办法。但我注意到你似乎假设权重（或初始化对象）按高斯分布初始化？这似乎有点受限。宽网络方法的吸引力，我认为在于任何独立的权重初始化都会导致 GP 的变换。也许我误解了含义。即使最后一层权重不是正态，你通常能把依赖追溯到其分布吗？还是说你需要 GP 来做条件化？一方面，高斯初始化并不真正涵盖实际使用的“所有架构”；更具体地说，这个研究纲领的最终目标似乎是推广到高斯之外（就像回归中用 CLT 正是为了摆脱参数假设）。还是说这是你计划的方向？
zwaps查看英文原文
Good results often seem "obvious" in ex post. So take this as a compliment. Intuitively (I have never read a paper in this field), since you are talking about wide networks, I also expected that a CLT would be used. For "dense" layers it is pretty obvious that one should be able to characterize each layer aggregation based on a CLT, and so forth. Some sort of mild independence assumption, mixing or martingales, on the sampling should be sufficient. I think therefore the goal of a paper for any architecture would be to figure out a way to generalize this for different layers. However, one thing I notice is that you seem to assume that weights (or whatever is initialized) are initialized with a Gaussian distribution? That seems a bit restricted. The appeal of this approach with wide networks, I think, is that any independent initialization of weights would lead to transformations of GP. Perhaps I am misunderstanding also the implications. Could you generally trace a dependence onto the distribution of the last layer weights, even if they are not normal? Or do you need the GP for your conditioning? On the one hand, Gaussian initialization is I think not really encompassing "all architectures" as practically used, but more specifically, it seems that the end goal of this research program would be to generalize beyond this (much like in regression, one uses CLTs exactly to get away from parametric assumptions). Or is that where you plan to go?
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2019Hacker News 讨论帖144 分 · 34 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选