系统与数据库2021sigops.org
不计数（不可靠）的核心
Cores that don’t count
进阶 · 约 45 分钟读完前置知识：计算机体系结构基础、数据中心/分布式系统概念、数值计算基础
导读
这篇论文（来自 Google 的 HotOS 2021 文章）研究数据中心中“不计数的核心”（mercurial cores）问题：某些 CPU 核心会静默产生错误的计算结果，且难以被常规检测发现。作者指出这类核心在数千台机器中就会出现几例，并讨论了在超大规模数据中心里如何应对这一挑战。
HN 关注点：是因为“静默出错的 CPU 核心”直接威胁计算正确性，而 Google 的一手数据和经验非常罕见，讨论焦点在于故障率之高以及为何厂商不与用户合作解决。
HN 评论精选
这几条评论里：质疑 1 补充 2 亲历 1
亲历亲历者证实坏核心导致的静默错误在 ML 训练中真实存在且极难排查
我在 Google 做这个问题做了一年，非常有趣。我负责的子领域是跑机器学习训练的加速器（如 GPU）。用户经常报告训练中出现 NaN——梯度爆炸导致任务崩溃。有时是特定样本或模型开发者的数值错误，但有时是坏核心导致的（矩阵乘法、embedding 查找、向量运算等环节出错）。ML 通常对少量噪声相当宽容（尤其是统计性质好的噪声），有些训练任务能在大量未纠正、未检测的错误下照常完成。这个领域很难做，因为你很难判断 NaN 是模型的问题还是芯片的问题。
dekhn查看英文原文
I worked on this problem for the past year at Google. It's a fascinating problem. In my subarea I focused on accelerators (like GPUs) running machine learning training. Many users report problems like "NaN" during training- at some point, the gradients blow up and the job crashes. Sometimes these are caused by specific examples, or numerical errors on the part of the model developer, but sometimes, they are the result of errors from bad cores (during matrix multiplication, embedding lookup, vector op, whatever). ML is usually pretty tolerant of small amounts of added noise (especially if it's got nice statistcal properties) and some training jobs will ride through a ton of uncorrected and undetected errors with few problems. It's a very challenging field to work in because it's hard to know if your nan is because of your model or your chip.
质疑质疑故障率其实不低，且论文未提及与厂商合作排查
网站挂了，存档链接：[archive.org]。让我印象深刻的是：论文说“mercurial 核心极其罕见”，但又说“每几千台机器就能观察到几个 mercurial 核心”。平均每 1000 台机器就有一个坏核心？这比例相当高。厂商肯定知道这个问题吧？就算没测出来，自家服务器也该遇到过。我通读了全文，没看到他们提到联系过厂商。双方都有强烈动机去解决或缓解这个问题，为什么不合作呢？
guidovranken查看英文原文
Site is down, archive link: https://web.archive.org/web/20210602080638/https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s01-hochschild.pdf What stands out to me: - "Mercurial cores are extremely rare" but "we observe on the order of a few mercurial cores per several thousand machines". On average one core per 1000 machines is faulty? That's quite a high rate. - Vendors surely must know about this? If not by testing then through experiencing the failures in their company servers. - I've read the whole paper and I see no mention of them even reaching out to vendors about this issue. Their are strong incentives on both sides to solve or mitigate this issue so why aren't they working together?
补充补充一个具体的坏核心根因分析案例
文章引用了 Dixit 等人对一个 CEE（缓存一致性错误）根因的调查，值得一读：[arxiv 链接]。迭代几次后发现，Scala 中 math.pow 函数的输入 Int(1.153)=0 在 CPU 的 Core 59 上总是返回 0，但换成 Int(1.152)=142 结果就是正确的。
freehrtradical查看英文原文
The article references Dixit et al. for an example of a root cause investigation of a CEE which is an interesting read: https://arxiv.org/pdf/2102.11245.pdf > After a few iterations, it became obvious that the computation of 𝐼𝑛𝑡(1.153)=0 as an input to the math.pow function in Scala would always produce a result of 0 on Core 59 of the CPU. However, if the computation was attempted with a different input value set 𝐼𝑛𝑡(1.152)=142 the result was accurate.
补充HPC 从业者希望论文公开更多故障部件细节，并指出厂商侧排查是黑箱
我希望看到更多关于缺陷部件的细节，尤其是 CPU 型号的统计（匿名也行）以及架构中哪个部分出现故障。做 HPC 时我处理过 FMA 单元算错结果、随机出现 NaN 之类的报告。要不是我们知道这种事可能发生、加上客户对自己代码的深入了解，真不敢想正常运维要怎么排查。坏部件会退回 CPU 厂商，进一步测试通常确认故障，但那之后的流程对厂商以外的人完全是黑箱。我也很想了解更多。
gbrown_查看英文原文
I'd love to see more details on the defective parts, particularly counts of CPU model (anonymized if needs be) and counts of which part of the architecture exhibited faults. From working in HPC I've handled reports of things like FMA units producing incorrect results or random appearance of NaNs. Were it not for the fact that we knew these things could happen and customer's intimate knowledge of their codes I dread to think how'd "normal" operations would track these issues down. Bad parts went back to the CPU manufacturer and further testing typically confirmed the fault. But that end of the process was pretty much a black box to anyone but the CPU manufacturer. I'd be keen to know more about this too.
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
- 视频与演讲
- 评论区提到的论文
历次讨论
- 2021Hacker News 讨论帖178 分 · 71 评
- 2024Hacker News 讨论帖106 分 · 13 评
同领域还值得读
I/O 比 CPU 更快——让我们划分资源并消除操作系统抽象
I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions
指出在现代服务器硬件上 I/O 速度已接近甚至超过 CPU，传统操作系统抽象反而成为瓶颈。
系统与数据库2019 · 817 分 · 277 评进阶 · 约 30 分钟编辑精选使用 io_uring 的高效 I/O
Efficient IO with io_uring
由 Linux 内核开发者 Jens Axboe 撰写，介绍 Linux 新的异步 I/O 接口 io_uring。
系统与数据库2019 · 426 分 · 96 评进阶 · 约 45 分钟编辑精选线程对大多数用途而言是个坏主意（1995）
Threads Are a Bad Idea for Most Purposes (1995)
这篇 1995 年的文章由 John Ousterhout 撰写，讨论为什么线程对大多数用途而言是坏主意，并主张事件驱动编程通常更简单、更高效。
系统与数据库2017 · 423 分 · 382 评进阶 · 约 25 分钟编辑精选学习型索引结构之论证
The Case for Learned Index Structures
提出把索引看作一种"模型"：既然 B-tree 等结构本质上是在学习数据的分布，就可以用机器学习模型（如回归模型、神经网络）来近似键到位置的映射，从而替代 B-tree、哈希索引…
系统与数据库2017 · 398 分 · 66 评 · 引用 25进阶 · 约 60 分钟编辑精选FoundationDB：一个分布式、解耦的事务型键值存储
FoundationDB: A distributed, unbundled, transactional key value store
介绍 FoundationDB：一个分布式、解耦设计的事务性键值存储。
系统与数据库2021 · 283 分 · 99 评进阶 · 约 90 分钟编辑精选