AI 与机器学习2024arxiv.org编辑精选
Mixtral 8x7B：稀疏混合专家语言模型
Mixtral 8x7B: A sparse Mixture of Experts language model
进阶 · 约 50 分钟读完前置知识：Transformer 基础、混合专家（MoE）、语言模型评测
导读
Mistral AI 发布的 Mixtral 8x7B 是一个稀疏混合专家（MoE）语言模型，每层由 8 个专家组成，每个 token 只激活其中 2 个，因此推理时实际参数量约 13B，但总参数量达 46.7B。论文称其性能在多数基准上优于或匹敌 Llama 2 70B 和 GPT-3.5，且支持 32k 上下文。
HN 关注点：是当时最强的开源可本地部署模型之一，讨论焦点集中在其实际使用体验、参数效率（13B 激活量）以及专家可能由 Mistral 7B 演化而来的发现。
HN 评论精选
这几条评论里：支持 1 质疑 1 补充 1 亲历 1
支持Mixtral 激活参数仅 13B，消费级显卡即可流畅运行，性能超 GPT-3.5，适合本地部署应用。
值得注意的是，这个模型的参数使用量足够低（约 13B），可以在 3090 上以高质量流畅运行，同时在 HumanEval 上超过 GPT-3.5，还支持 32k 上下文。3090 是消费级显卡，游戏玩家中很常见。我希望游戏开发者开始尝试在游戏中本地部署 Mixtral，比如做一个每个领袖都由 LLM 驱动的《文明》。
cuuupid查看英文原文
I’d like to note that this model’s parameter usage is low enough (13b) to run smoothly at high quality on a 3090 while beating GPT-3.5 on humaneval and sporting 32k context. 3090s are consumer grade and common on gaming rigs. I’m hoping game devs start experimenting with locally deployed Mixtral in their games. e.g. something like CIV but with each leader powered via LLM
亲历亲历者确认模型实际表现优异，并指出专家很可能由 Mistral 7B 种子分化而来，训练成本远低于预期。
这篇论文介绍的是已经发布约一个月的模型。Mixtral 8x7B 非常出色，实际激活参数约 13B，在社区排行榜上远超同规模模型。它比基准测试表现还要好一点，作为 Mac 上的日常模型（聊天、代码等）能力惊人。权重发布后（论文未提及）的一个发现是：8 个专家很可能都是由 Mistral 7B 作为种子初始化后各自分化而来。这引发了本地 LLM 社区大量关于“克隆模型来廉价生成专家”的实验。此前普遍认为训练 8x7B 网络的工作量相当于训练 8 个 7B 模型，但对 Mistral 来说似乎并非如此，这非常有趣。这个领域创新很快，DeepMind 本周也发布了 Calm 论文，我认为今年 LLM 架构上会有很多有趣的改进。
vessenes查看英文原文
This paper details the model that's been in the wild for approximately a month now. Mixtral 8x7B is very, very good. It's roughly sized at 13B, and ranked much, much higher than competitively sized models by, e.g. https://www.reddit.com/r/LocalLLaMA/comments/1916896/llm_comparisontest_confirm_leaderboard_big_news/. Ravenwolf notes that the model slightly outperforms some of its benchmark testing, and this is my experience. It's surprisingly good for a model of its size, and a very capable daily driver on a Mac for chat, code input and other uses. Something that has come to light since the release of the weights, and not mentioned in this paper is that it looks like fairly likely that the 8 experts were all seeded by Mistral 7B and subsequently diverged. This has generated a lot of experimentation in the local LLM community with cloning models as a way to cheaply generate experts. It was generally thought likely that training an 8x7B network would be as much work as training 8 7B networks, but this seems not to have been true for Mistral, which is super interesting. There's still a lot of rapid innovation happening in this space, with papers like Calm from DeepMind this week, and a lot of the adhoc experimental layer combining happening in the wild, (see, e.g. Goliath-120b), I think we're likely to see some pretty interesting architectural improvements this year in the LLM space. Calm seems to point the way to a next step after MoE, and models like Goliath seem to indicate that even a really really lazy version of Calm (no Linear layer combination, just literally alternating layers at full weights) can be very impactful. Overall I think we will see really, really strong models that are performant on consumer hardware in 2024, likely first half of this year.
质疑质疑论文只是补发已发布一个月的模型的技术报告。
这不就是去年年底已经发布过的那个模型吗？
smcleod查看英文原文
Wasn’t this what was released at the end of last year?
补充询问本地运行 Mixtral 的便捷工具和方法。
这个模型能用 Simon Willison 的 LLM 工具运行吗？我在 issues 和讨论里都没找到关于 Mixtral 的提及。除了那个工具，还有没有简单的命令行方式来体验这个模型？
xrd查看英文原文
Is this a model that can be run using Simon Wilison's LLM tool? I cannot find any mention of Mixtral in the issues nor in the discussions. Is there an easy way to play with this model from the command line other than that?
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2024Hacker News 讨论帖359 分 · 150 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选