AI 与机器学习2025arxiv.org
深度学习并不那么神秘或不同
Deep Learning Is Not So Mysterious or Different
进阶 · 约 60 分钟读完前置知识：机器学习基础、过拟合与正则化、假设空间与归纳偏置
导读
这篇论文（arXiv 2503.02113）主张深度学习并不像通常认为的那样神秘或独特：它并非通过把假设空间限制得很小来防止过拟合，而是采用一个灵活的假设空间，并对与数据一致的更简单解给予“软偏好”。作者认为这一原则可以编码进多种模型类别，因此深度学习与其他模型类别并没有本质区别。
HN 关注点：HN 读者围绕“深度学习是否真的特殊”展开争论：有人追问这种软偏好具体如何实现，有人质疑“黑箱/神秘”本身就是伪命题，也有人由此联想到用统计方法复刻语言模型的可行性。
HN 评论精选
这几条评论里：支持 1 质疑 1 补充 2
补充设想用词距共现统计加频率查找的朴素方法，检验它离 GPT-2 有多远。
我希望有时间试试这个：1) 抓取大量 GB 级的文本（书籍等）；2) 对每个词、其后的每个第 N 个词，记录与当前词的距离，并累加该词对/距离的计数；3) 对每个词，存下每个距离上出现最频繁的词；4) 做一个预测算法，根据用户输入决定下一个输出词（或词集），本质上是比较词对/距离，找出最可能的下一个词。这能有多接近 GPT-2？还可以更进一步：每个距离存多个词，按频率排序。
YesBox查看英文原文
I wish I had the time to try this: 1.) Grab many GBs of text (books, etc). 2.) For each word, for each next $N words, store distance from current word, and increment count for word pair/distance. 3.) For each word, store most frequent word for each $N distance. [a] 4.) Create a prediction algorithm that determines the next word (or set of words) to output from any user input. Basically this would compare word pairs/distance and find most probable next set of word(s) How close would this be to GPT 2? [a] You could go one step further and store multiple words for each distance, ordered by frequency
补充推荐斯坦福概率论课程和 3b1b 作为机器学习与深度学习的入门资源。
如果有人想深入机器学习，我找到的极好资源之一是斯坦福的《Probability for computer scientists》。它深入讲解概率论和机器学习的理论基础，在我看来比我看过的任何课程都好。（Andrew Ng 当然是传奇，但他的课要求对线性代数有些熟悉。）至于深度学习，3b1b 很适合做可视化入门。
rottc0dd查看英文原文
If anyone wants to delve into machine learning, one of the superb resources I have found is, Stanfords "Probability for computer scientists"(https://www.youtube.com/watch?v=2MuDZIAzBMY&list=PLoROMvodv4rOpr_A7B9SriE_iZmkanvUg). It delves into theoretical underpinnings of probability theory and ML, IMO better than any other course I have seen. (Yeah, Andrew Ng is legendary, but his course demands some mathematical familarity with linear algebra topics) And of course, for deep learning, 3b1b is great for getting some visual introduction (https://www.youtube.com/watch?v=aircAruvnKk&list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi).
质疑质疑论文的“软偏好简单解”说法：深度学习到底如何实现复杂度惩罚，与经典惩罚似然有何不同？
论文说：不通过限制假设空间来避免过拟合，而是拥抱灵活的假设空间，对与数据一致的更简单解给予软偏好；这一原则可编码进多种模型类别，因此深度学习并不像看起来那么神秘或独特。深度学习是怎么做到这一点的？我上次深入接触机器学习时用的是惩罚似然：在模型空间上优化一个代价函数，它等于两项之和，一项衡量模型预测与数据的差异，另一项衡量模型复杂度。这正好编码了“对与数据一致的更简单解的软偏好”，但深度学习是这样运作的吗？我印象中深度学习惩罚复杂度的方式更复杂、没那么直接。
getnormality查看英文原文
> rather than restricting the hypothesis space to avoid overfitting, embrace a flexible hypothesis space, with a soft preference for simpler solutions that are consistent with the data. This principle can be encoded in many model classes, and thus deep learning is not as mysterious or different from other model classes as it might seem. How does deep learning do this? The last time I was deeply involved in machine learning, we used a penalized likelihood approach. To find a good model for data, you would optimize a cost function over model space, and the cost function was the sum of two terms: one quantifying the difference between model predictions and data, and the other quantifying the model's complexity. This framework encodes exactly a "soft preference for simpler solutions that are consistent with the data", but is that how deep learning works? I had the impression that the way complexity is penalized in deep learning was more complex, less straightforward.
支持反对“软件/模型是黑箱或神秘”的说法，认为机器状态在原理上完全可确定。
暗示任何软件是“神秘的”都是有问题的——这里没有什么玄乎的东西——运行软件的机器状态在每个周期都可以被精确确定，执行的指令、数据以及下一条指令也都可以精确确定。任何软件是“黑箱”的说法，不过是广告术语，是那些想把自己当成《黑客军团》里祭司阶层的技术兄弟们在传播。
EncomLab查看英文原文
The implication that any software is "mysterious" is problematic - there is no "woo" here - the exact state of the machine running the software may be determined at every cycle. The exact instruction and the data it executed with may be precisely determined, as can the next instruction. The entire mythos of any software being a "black box" is just so much advertising jargon, perpetuated by tech bros who want to believe they are part of some Mr. Robot self-styled priestly class.
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2025Hacker News 讨论帖485 分 · 126 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选