AI 与机器学习2024arxiv.org
神经网络可训练性的边界是分形的
The boundary of neural network trainability is fractal
进阶 · 约 45 分钟读完前置知识：神经网络训练基础、超参数调优经验、混沌系统基本概念
导读
这篇论文通过大量小规模神经网络训练实验，绘制了“可训练性”随超参数（如学习率、初始化种子等）变化的图像，发现可训练与不可训练区域的边界呈现出分形（自相似）结构。作者将这种混沌边界与神经网络训练的不可预测性联系起来。
HN 关注点：HN 读者对“训练成功与否本质上是混沌/分形现象”这一反直觉且视觉冲击力强的结论很感兴趣，讨论它是否显而易见、以及在实际训练场景中是否成立。
HN 评论精选
这几条评论里：质疑 2 补充 2
补充猜测混沌边界与大脑的递归动力学相关，可能指向更高效的 AI 训练方式
Wolfram 曾指出语言语义中的内部结构比我们预想的要多得多，与乔姆斯基的观点相反。我们也知道大脑（尤其是皮层）高度递归，因此天然适合产生分形和混沌混合。也许隐藏的结构就是让某组神经元进入分形/混沌振荡所需的神经超参数——这比快速收敛到不动点的配置包含多得多的信息量。或许深度神经网络的学习正是在制造一种处于临界点的条件，以获得高信息量的生成状态，再用大量自由参数将其塑形以拟合目标系统。这暗示使用更接近真实神经元动力学的迭代生成器做 AI 会更高效，就像匹配物理系统中的谐振器一样。
pmayrgundter查看英文原文
One of Wofram's comments is that there appears to be much more internal structure in language semantics that we had expected, contra-Chomsky. We also know the brain, cortex esp, is highly recurrent, so it should be primed for creating fractals and chaotic mixing. So maybe the hidden structure is the set of neural hyperparams needed to put a given cluster of neurons into fractal/chaotic oscillations like this. Seems potentially more useful too.. way more information content than a configuration that yields a fast convergence to a fixed point. Perhaps this is what learning deep NNs is doing: producing conditions where the substrate is at the tipping point, to get to a high-information generation condition, and then shaping this to fit the target system as well as it can with so many free parameters. That suggests that using iterative generators that are somehow closer to the dynamics of real neurons would be more efficient for AI: it'd be easier to drive them to similar feedback conditions and patterns Like matching resonators in any physical system
质疑质疑实验依赖固定随机种子，混沌边界在实际训练场景中是否成立尚待验证
这个玩具实验很有趣。我注意到每张图都使用相同的权重初始化，这大概使像素之间更平滑；我猜训练（数据打乱）也用了相同的随机种子。我很好奇如果每个像素用不同的随机性/打乱顺序，图会是什么样——高学习率下可能太噪，但在更常见、更实用的学习率下或许能看到分形行为。也可以对每个数据集的随机初始化做同样的事。这能回答混沌边界是否也存在于更实际的使用场景中。
mchinen查看英文原文
Reposting comment from last time since I'm still curious: This is really fun to see. I love toy experiments like this. I see that each plot is always using the same initialization of weights, which presumably makes it possible to have more smoothness between each pixel. I also would guess it's using the same random seed for training (shuffling data). I'd be curious to know what the plots would look like with a different randomness/shuffling of each pixel's dataset. I'd guess for the high learning rates it would be too noisy, but you might see fractal behavior at more typical and practical learning rates. You could also do the same with the random initialization of each dataset. This would get at if the chaotic boundary also exists in more practical use cases.
质疑认为分形边界这一结论并不令人意外
我不是想贬低这篇论文的贡献，但这种情况难道不是显而易见的吗？
xcodevn查看英文原文
I am not trying to downplay the contribution of the paper, but isn't it obvious that this is the case?
历次讨论
- 2024Hacker News 讨论帖200 分 · 65 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选