AI 与机器学习2025arxiv.org
通用权重子空间假设
The universal weight subspace hypothesis
进阶 · 约 60 分钟读完前置知识：Vision Transformer 基础、模型微调与 LoRA、线性代数（子空间投影）
导读
论文提出“通用权重子空间”假设：大量微调模型的权重可能共享一个很低维的共同子空间。作者将 ViT 权重投影到识别出的 16 维通用子空间后重建，报告分类精度没有显著下降，并能用单个通用子空间模型表示数百个 ViT，显著降低内存。评论还提到该工作在 LLaMA3-8B、GPT-2、Flan-T5 等模型上观察到类似共性。
HN 关注点：是否意味着大量微调模型可被压缩/合并进一个低维子空间，以及所谓 universal 是否只是同一基座模型微调结果彼此相似。
HN 评论精选
这几条评论里：支持 1 质疑 1 补充 2
支持论文关键结果是 16 维通用子空间可重建未见 ViT 而不掉点，并能用一个子空间模型替代数百个微调 ViT，大幅省内存。
对想抓住论文重点的人，我认为最关键的是文中中段连续两段里的两句话：“我们另外选了五个此前未见、且能获得评估数据的预训练 ViT 模型。这些模型相对初始集合属于域外；我们把它们的全部权重投影到所识别的 16 维通用子空间上重建，然后评估分类准确率，未发现性能显著下降。”以及“我们可以用单个 Universal Subspace 模型替代这 500 个 ViT 模型。忽略随任务变化的第一层和最后一层……我们看到内存需求减少 100 倍，而且随着训练模型数量增加，这种节省还会扩大。据我们所知，这是首个能把 500 个（理论上更多）Vision Transformer 合并进单个通用子空间模型的工作。这意味着数百个 ViT 可以由一个子空间模型表示。”所以，他们发现 50 个 LLaMA3-8B、177 个 GPT-2 和 8 个 Flan-T5 模型训练后的结构存在共同点；证明该共同点可替换原模型中的对应结构而不损失功能；并称首次发现这一点。
altairprime查看英文原文
For those trying to understand the most important parts of the paper, here's what I think is the most significant two statements, subquoted out of two (consecutive) paragraphs midway through the paper: > we selected five additional, previously unseen pretrained ViT models for which we had access to evaluation data. These models, considered out-of-domain relative to the initial set, had all their weights reconstructed by projecting onto the identified 16-dimensional universal subspace. We then assessed their classification accuracy and found no significant drop in performance > we can replace these 500 ViT models with a single Universal Subspace model. Ignoring the task-variable first and last layer [...] we observe a requirement of 100 × less memory, and these savings are prone to increase as the number of trained models increases. We note that we are, to the best of our knowledge, the first work, to be able to merge 500 (and theoretically more) Vision Transformer into a single universal subspace model. This result implies that hundreds of ViTs can be represented using a single subspace model So, they found an underlying commonality among the post-training structures in 50 LLaMA3-8B models, 177 GPT-2 models, and 8 Flan-T5 models; and, they demonstrated that the commonality could in every case be substituted for those in the original models with no loss of function; and noted that they seem to be the first to discover this. For a tech analogy, imagine if you found a bzip2 dictionary that reduced the size of every file compressed by 99%, because that dictionary turns out to be uniformly helpful for all files. You would immediately open a pull request to bzip2 to have the dictionary built-in, because it would save everyone billions of CPU hours. [*] [*] Except instead of 'bzip2 dictionary' (strings of bytes), they use the term 'weight subspace' (analogy not included here[**]) — and, 'file compression' hours becomes 'model training' hours. It's just an analogy. [**] 'Hilbert subspaces' is just incorrect enough to be worth appending as a footnote[***]. [***] As a second footnote.
质疑质疑 universal 的范围：可能只是同一基座模型的众多微调/LoRA 彼此相似，且部分模型来源不可靠。
这说法有点混乱。他们说“500 个 Vision Transformer”时，实际指的是同一个基座模型的 500 个微调版本，从匿名网友的 Hugging Face 账号下载。据我所知，这些空间只对单个预训练基座模型才是“通用”的。同一基座的微调模型彼此极其相似，尤其是 LoRA，这真的那么令人惊讶吗？我访问了他们引用的一个模型，Hugging Face 显示它含有恶意软件：https://huggingface.co/lucascruz/CheXpert-ViT-U-MultiClass
modeless查看英文原文
This seems confusingly phrased. When they say things like "500 Vision Transformers", what they mean is 500 finetunes of the same base model, downloaded from the huggingface accounts of anonymous randos. These spaces are only "universal" to a single pretrained base model AFAICT. Is it really that surprising that finetunes would be extremely similar to each other? Especially LoRAs? I visited one of the models they reference and huggingface says it has malware in it: https://huggingface.co/lucascruz/CheXpert-ViT-U-MultiClass
补充若模型自然收敛到通用低维空间，可据此初始化或约束训练以节省算力。
有意思……如果模型天然会收敛到一个通用低维空间，这可能让训练快得多：你可以把初始化或训练约束在这个空间内，而不是每次都花大量算力从头重新发现它。
kacesensitive查看英文原文
interesting.. this could make training much faster if there’s a universal low dimensional space that models naturally converge into, since you could initialize or constrain training inside that space instead of spending massive compute rediscovering it from scratch every time
补充提议用遗传算法搜索和改进这类子空间结构。
我总想让遗传算法被用来开发并改进这些结构……不过在任何神经网络讨论里，我都希望遗传算法能出现……
VikingCoder查看英文原文
I find myself wanting genetic algorithms to be applied to try to develop and improve these structures... But I always want Genetic Algorithms to show up in any discussion about neural networks...
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2025Hacker News 讨论帖358 分 · 132 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选