AI 与机器学习2024arxiv.org
自压缩神经网络
Self-Compressing Neural Networks
进阶 · 约 30 分钟读完前置知识：深度学习基础、模型压缩、优化目标与损失函数
导读
这篇论文研究让神经网络在训练过程中自我压缩：把内存和计算量（FLOPS）作为额外的优化目标，让网络自己学习精简结构，而不是由人预先设定模型大小。评论者视其为神经网络自我优化的早期尝试，但距离生物大脑那种修改拓扑结构的神经可塑性机制还很远。
HN 关注点：两点：这种「自组织/自我优化」是否是类脑网络的起点，以及一个看似显而易见、一年半前就发表的想法为何没有流行起来。
HN 评论精选
这几条评论里：支持 1 质疑 1 补充 1 亲历 1
补充视其为类脑自我优化网络的早期一步，但离真正的神经可塑性机制仍很远。
我认为这可能是让神经网络真正模仿生物大脑的第一步。在我看来，NN 架构中缺失的最大一块是类似神经可塑性的机制，能修改神经元的拓扑结构——大脑会围绕所学的东西重新组织自己。这篇论文距离实现突触修剪/强化/弱化、神经发生或突触发生还很远，但这是我见过的第一个网络进行自我优化的例子。
throwup238查看英文原文
I think this might be the first step to making neural networks that actually mimic biological brains. IMO the biggest piece missing from NN architectures is a mechanism like neuroplasticity that modifies the topology of neurons. Brains reorganize themselves around the things they learn. This paper is a long way from implementing synaptic pruning/strengthening/weakening, neurogenesis, or synaptogenesis but it’s the first one I’ve seen where the network is self optimizing.
支持惊讶于这个想法来得这么晚；若在大模型上有效，损失函数可能纳入硬件执行时间等具体指标。
这非常酷。令我惊讶的是花了这么久才有人尝试，回头看这似乎是如此显而易见的想法——当然，有人想出来之后这么说很容易。如果这在更大的模型上也能奏效，我们可能会看到损失函数纳入越来越具体的性能指标，甚至可以想象纳入特定硬件上的实际执行时间。
Version467查看英文原文
This is super cool. It's surprising to me that it took so long for someone to try this. It seems like such an obvious idea (in hindsight). But I guess that's easy to say now that someone came up with it. If this turns out to work well even for much larger models, then we might see loss functions that incorporate ever more specific performance metrics, conceivably even actual execution times on specific hardware.
质疑质疑为何论文发表一年半仍未流行。
所以这是一年半前发表的？有没有什么原因让它没有流行起来？
spacemanspiff01查看英文原文
So this was published a year and a half ago? Is there a reason it did not catch on?
亲历希望模型不预设固定大小、能随训练自组织缩放，并联系到自己做的 Self-Organizing Gaussian Splats。
把尽可能少的计算资源（内存和/或 FLOPS）作为训练 NN 的额外优化标准，是条有趣的路径。我觉得当前预训练模型家族的状态很奇怪：以 Llama 3.1 或 Segment Anything 2 为例，tiny/small/medium/larger/huge 每个层级的大小都是预先定义的，而且多少是独立训练的，这感觉很别扭、零散，像是还没真正到位。我想要一个能根据推理时的任务上下缩放、训练开始时没有固定大小的模型——它不该在训练中看到更多 token 时自我专门化、在需要的地方增长吗？而不需要人预先定死大小。自组织对我来说是个迷人的话题，过去一年我在做 Self-Organizing Gaussian Splats，眯着眼看，它和这篇论文处于相似的空间：用自组织映射（90 年代的好概念）以内存高效的方式表示 3D 场景。
w-m查看英文原文
Using as little computational resources (memory and/or FLOPS) as possible as an additional optimization criterion when training NNs is an interesting avenue. I think the current state of pre-trained model families is weird. Take Llama 3.1 or Segment Anything 2: you get tiny/small/medium/larger/huge models, where for each tier the model size was predefined, and they are trained somewhat (completely?) independently. This feels iffy, patchy, and like we haven't really arrived yet. I'd want a model that scales up and down depending on the task given at inference, and a model that doesn't have a fixed size when starting the training. Shouldn't it specialize over training progress, when seeing more tokens, and grow larger where needed? Without some human fixing a size beforehand? Self-organization is a fascinating topic to me. This last year I've been working on Self-Organizing Gaussian Splats [0]. With a lot of squinting, this lives in a similar space as the Self-Compressing Neural Networks from the link above. The idea of the Gaussians was to build on Self-Organizing Maps (lovely 90s concept, look for some GIFs if you don't know it), and use that to represent 3D scenes in a memory-efficient way. By mapping attributes into a locally smooth 2D grid. It's quite a simple algorithm, but works really well, and better than many quite complicated coding schemes. So this has me excited that we'll (re-)discover great methods in this space in the near future. [0]: https://fraunhoferhhi.github.io/Self-Organizing-Gaussians/
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2024Hacker News 讨论帖233 分 · 57 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选