网络与分布式2021sigops.org
分布式系统中的亚稳态故障
Metastable Failures in Distributed Systems
进阶 · 约 25 分钟读完前置知识：分布式系统基础、可用性与故障模型、反馈控制基础
导读
这篇 HotOS 2021 论文提出并系统化了「亚稳态故障」（metastable failure）这一概念：系统在负载超过某个隐形的阈值后进入「脆弱状态」，此时运行正常甚至更高效，但一旦被某个触发事件推入故障，反馈回路会自我维持，使系统停留在「稳定但不可用」的状态，直到有足够大的外部纠正动作才恢复。论文给出这类故障的模型、案例与分类，并主张用控制理论视角来设计和运维分布式系统，以避免这种多稳态陷阱。
HN 关注点：它把「系统稳定但不干活」这种隐蔽的严重故障模式讲清楚了，并引发用控制理论治理大型系统的讨论。
后来怎么样
该文提出并系统化的“亚稳态故障”已成为分布式系统可靠性讨论中的常用概念，被用于分析自维持故障与脆弱状态，并推动了用控制理论视角设计和运维系统的讨论。
HN 评论精选
这几条评论里：支持 1 补充 2
支持论文把「稳定但不可用」的多稳态故障讲清楚了，价值在于表述与证据，并呼吁用控制理论指导系统设计。
我认为这篇论文非常重要，任何设计或运维大型系统的人都该读一读并把核心观点记在心里。作为系统设计者，我们习惯把系统分成「稳定」和「不稳定」，稳定是好的、不稳定是坏的。这篇论文指出的是，很多分布式系统存在多个「稳定」模式，其中一些模式在控制理论意义上是稳定的，但从客户端角度看完全没在干有用的事。这很危险，因为除非有东西改变——人工介入、控制面采取行动等——系统不会自己从这个「稳定但宕机」的模式里跳出来。我不觉得论文里有特别新的东西，但以这种形式写下来、配上他们给出的证据，非常有价值。希望它能加深把控制理论应用于分布式系统设计与控制问题的讨论，让这类系统的设计能采用更理论化的方法，避开不稳定和双稳态的常见成因。作者之一在博客上有一篇很好的论文总结：http://charap.co/metastable-failures-in-distributed-systems/ 我也写了一篇总结和讨论：https://brooker.co.za/blog/2021/05/24/metastable.html
mjb查看英文原文
I think this paper is super important, and anybody who designs or runs big systems should read it and take the core point to heart. As system designers, we're very used to thinking about systems as 'stable' and 'unstable', where stability is good, and instability is bad. What this paper points out is that many kinds of distributed systems have multiple 'stable' modes, some of which are modes where the system is stable (in a control theory sense), but not doing any useful work from the client's perspective. This is dangerous, because the system won't kick itself out of this "stable but down" mode without something changing: human input, a control plane taking action, etc. I don't think this paper covers anything particularly new, but writing it down in this form, with the evidence they present, is very valuable. Hopefully this paper will deepen the conversation about applying control theory to distributed systems design and control problems, and allow a more theoretical approach to be taken to the design of these systems to avoid common causes of instability and bistability. One of the authors has a great summary of the paper on his blog: http://charap.co/metastable-failures-in-distributed-systems/ I wrote a summary and discussion too: https://brooker.co.za/blog/2021/05/24/metastable.html
补充亚稳态故障的框架可推广到供应链、虚假信息、金融与宏观经济等领域的持续性问题。
读这篇论文时我意识到，很多其他领域的持续性大规模系统性问题也可以用亚稳态故障来理解——比如持续的供应链问题、社交网络上持续的虚假信息浪潮、政府对金融机构的持续救助、持续的熊市和深度经济衰退。引用论文原文：「系统从稳定状态开始。一旦负载超过某个阈值——隐形的、不可见的——系统就进入脆弱状态。脆弱的系统是健康的，但可能因某个触发事件跌入无法恢复的亚稳态。脆弱状态不是过载状态；系统可以在脆弱状态下运行数月乃至数年，然后在负载毫无增加的情况下卡进亚稳态。事实上，很多……系统选择一直运行在脆弱状态，因为它的效率远高于稳定状态。当众多潜在触发事件之一使系统进入亚稳态后，一个反馈回路会维持故障，使系统停留在故障状态，直到施加足够大的纠正动作。在最严重的故障中，反馈回路具有传染性，会让未暴露于触发事件的系统部分也进入故障状态。涉及亚稳态故障的故障，最初常被归咎于触发事件，但真正的根因是系……」
cs702查看英文原文
As I was reading the paper, it dawned on me that persistent large-scale systemwide problems in many other domains can be understood as metastable failures -- including, for example, persistent supply chain problems, persistent waves of disinformation on social networks, persistent government support/rescue of financial firms, persistent bear markets, and deep economic recessions. Quoting from the paper: "A system starts in a stable state. Once the load rises above a certain threshold -- implicit and invisible -- the system enters a vulnerable state. The vulnerable system is healthy, but may fall into an unrecoverable metastable state due to a trigger. The vulnerable state is not an overloaded state; a system can run for months or years in the vulnerable state and then get stuck in a metastable state without any increase in load. In fact, many ... systems choose to run in the vulnerable state all the time because it has much higher efficiency than the stable state. When one of many potential triggers causes the system to enter the metastable state, a feedback loop sustains the failure, causing the system to remain in the failure state until a big enough corrective action is applied. In the most severe outages, the feedback loop is contagious, causing portions of the system that weren’t exposed to the trigger to enter the failure state as well. It is common for an outage that involves a metastable failure to be initially blamed on the trigger, but the true root cause is the sustaining effect. Metastable failures have a disproportionate impact on hyperscale distributed systems. ... The strength of many feedback loops is proportional to the scale, so they can slip past even a robust testing and deployment regime. The difference between the trigger and the sustaining effect makes it hard to discover the correct response, increasing the time to recovery. Shedding load as a corrective action can be a further source of disruption for users."
补充补充相关资料：灰色故障与差异性可观测性，与本 paper 的隐蔽故障主题相关。
还没人提到「灰色故障」（gray failure）：https://www.cs.jhu.edu/~huang/paper/grayfailure-hotos17.pdf 。摘要大意：云规模提供了替换故障组件所需的大量资源，但这只有在这些故障能被检测到时才有用。因此我们在云环境中看到的主要可用性事故和性能异常，往往由细微的底层故障引起，即灰色故障而非 fail-stop 故障。本文讨论我们在生产级云规模系统中对灰色故障的经验，展示其广泛影响与后果。我们还认为灰色故障的一个关键特征是「差异性可观测」：即使应用已受影响，系统的故障检测器也可能察觉不到问题。这一认识让我们相信，要最好地应对它们，应着力弥合不同组件对「什么算故障」的认知差距。
sitkack查看英文原文
No one has mentioned "gray failure", https://www.cs.jhu.edu/~huang/paper/grayfailure-hotos17.pdf > Cloud scale provides the vast resources necessary to replace failed components, but this is useful only if those failures can be detected. For this reason, the major availability breakdowns and performance anomalies we see in cloud environments tend to be caused by subtle underlying faults, i.e., gray failure rather than fail-stop failure. In this paper, we discuss our experiences with gray failure in production cloud-scale systems to show its broad scope and consequences. We also argue that a key feature of gray failure is differential observabil- ity: that the system’s failure detectors may not notice problems even when applications are afflicted by them. This realization leads us to believe that, to best deal with them, we should focus on bridging the gap between different components’ perceptions of what constitutes failure.
历次讨论
- 2021Hacker News 讨论帖10 分 · 0 评
- 2021Hacker News 讨论帖161 分 · 12 评
同领域还值得读
Paxos vs. Raft：分布式共识是否已达成共识？
Paxos vs. Raft: Have we reached consensus on distributed consensus?
重新审视了 Paxos 与 Raft 两种分布式共识算法之间的差异，试图在一个统一的框架下比较它们，并论证二者的相似程度远高于通常的认知，许多所谓的差别只是表述层面的。
网络与分布式2020 · 343 分 · 65 评 · 引用 10进阶 · 约 28 分钟编辑精选Stellar 协议：互联网级共识的联邦模型
Stellar Protocol: A Federated Model for Internet-Level Consensus (2016)
提出 Stellar Consensus Protocol（SCP），一种面向互联网级共识的联邦拜占庭协议模型。
网络与分布式2018 · 301 分 · 61 评专业 · 约 90 分钟编辑精选拜占庭将军问题（1982）
The Byzantine Generals Problem (1982)
这篇 1982 年论文提出并形式化了拜占庭将军问题：在分布式系统中，部分节点可能叛变或故障、通信不可完全信任时，忠诚节点如何达成一致行动。
网络与分布式2018 · 290 分 · 77 评进阶 · 约 75 分钟编辑精选Google 变差了吗？对搜索引擎中 SEO 垃圾的纵向调查
Is Google Getting Worse? A Longitudinal Investigation of SEO Spam in Search
对搜索引擎结果中的 SEO 垃圾内容做了纵向追踪：作者团队在一段时间内持续采集 Google、Bing、DuckDuckGo 等引擎针对商品评测类查询的结果，用分类器识别联盟营销…
网络与分布式2024 · 276 分 · 274 评入门 · 约 45 分钟编辑精选使 CRDT 能够容忍拜占庭故障
Making CRDTs Byzantine Fault Tolerant
Martin Kleppmann 等人的论文研究如何让 CRDT（无冲突复制数据类型）抵御拜占庭故障节点，即部分参与者恶意篡改数据时仍能保证一致性。
网络与分布式2022 · 266 分 · 40 评进阶 · 约 90 分钟编辑精选