网络与分布式2022web.stanford.edu
我们需要在数据中心替换 TCP
We need a replacement for TCP in the datacenter
进阶 · 约 25 分钟读完前置知识：TCP/IP 基础、拥塞控制、数据中心网络
导读
这篇论文讨论为什么 TCP 不适合作为数据中心传输协议：数据中心应用多以离散消息/RPC 交互，而 TCP 提供的是字节流模型，且其拥塞控制等假设与数据中心网络环境不匹配，因此作者主张为数据中心重新设计传输层。
HN 关注点：HN 读者围绕“到底该换协议还是用 RDMA/更快网卡解决”展开争论，并延伸到字节流 vs 消息流这一更普遍的抽象问题。
HN 评论精选
这几条评论里：支持 1 质疑 2
质疑底层协议长期停滞，且缺乏跨协议栈的可观测/调试机制；论文只覆盖了网络层。
我多年来一直认为底层协议现在简直是个笑话，但业界没人愿意投入改进——企业看不到“短期股东价值”，就安于现状。这篇论文好笑的地方在于：它谈的只是网络层问题，完全忽略了“端口号”作为服务标识的抽象已经失败，因为业界把 HTTP 当成了所有应用层协议的通用隧道封装。而且还不止后端问题！单说 TCP，我们至今没有任何办法把一个完整事务在协议栈各层的信息逐跳传递并返回，用于调试。理想情况下应能把栈的每一层转发到每一跳再带回，从而程序化地定位网络问题、自动诊断并告诉用户怎么解决。但现在还得让人跳到用户电脑上，用一堆随机工具做神秘组合、靠人类直觉去猜发生了什么，活像网络界的甘道夫。40 多年了还这样。
throwaway892238查看英文原文
Yes!!! I have been saying for years that lower level protocols are a bad joke at this point, but nobody in the industry wants to invest in making things better. There are so many improvements we could be making, but corporations don't see any "immediate shareholder value", so they sit around happy as pigs in shit with the status quo. What's kind of hilarious about this paper is, these are just the network-layer problems! It completely ignores that the "port number" abstraction for service identification has completely failed due to the industry glomming onto HTTP as some sort of universal tunnel encapsulation for all application-layer protocols. And then there's all the non-backend problems! And that's just TCP. We still lack any way to communicate up and down the stack of an entire transaction, for example for debugging purposes. We should have a way to forward every single layer of the stack across each hop, and return back each layer of the stack, so that we can programmatically determine the exact causes of network issues, automatically diagnose them, and inform the user how to solve them. But right now, you need a human being to jump onto the user's computer and fire up an assortment of random tools in mystical combinations and use human intuition to divine what's going on, like a god damn Networking Gandalf. And we've been doing it this way for 40+ years.
质疑现有 RDMA/InfiniBand 和高速网卡已能解决问题，替换 TCP 成本高、收益有限。
我理解这种出发点，但不认同：数据中心不需要替换 TCP。为什么？因为低延迟、严格流控或追求 99.99% 利用率的场景本来就不用 TCP（存储是高吞吐、低延迟且有严格流控，也不用 TCP，NFS 和 iSCSI 除外）。如果这真是大问题，大家早该转向 InfiniBand 上的 RDMA 了；共享内存集群多年来就是这么做的，但通用计算中很少见，多数时候不值得费劲。现在 InfiniBand 便宜了，部署 RDMA 类互连并不难；可靠的二层加内置流控能解决不少问题，哪怕上面继续跑 IP。25/100G 也便宜了，多数问题靠给服务器加网卡、在机架/核心网上做分布式 LACP 之类就能解决。最大的问题往往不是网络限制吞吐，而是处理或其他非网络 IO。
KaiserPro查看英文原文
I get where this is coming from, but no. We don't need to replace TCP in the datacentre. Why? because for things that are low latency, need rigid flow control, or other 99.99% utilisation case, one doesn't use TCP. (Storage, which is high throughput, low latency and has rigid flow control, doesn't [well ignore NFS and iscisi] use TCP) Look if it really was that much of a problem then everyone in datacentres would move to RDMA over infiniband. For shared memory clusters, thats what's been done for years. but for general purpose computing its pretty rare. Most of the time its not worth the effort. Infiniband is cheap now, so its not that hard to deploy RDMA[1] type interconnects. Having a reliable layer2 with inbuilt flow control solves a number of issues, even if you are just slamming IP over the top. shit, even 25/100gig is cheap now. so most of your problems can be solved by putting extra nics in your servers and have fancypants distributed LACP type setups on your top of rack/core network. The biggest issue is that its not the network that's constraining throughput, it either processing or some other non network IO. [1]I mean it is hard, but not as hard as implementing a brand new protocol and expecting it to be usable and debuggable.
支持字节流抽象不适合大多数应用，消息分帧被每个协议重复实现，应转向消息流模型。
这不只是数据中心的问题。我创建或实现过的每个网络协议——每个消息系统、每个视频游戏、每个 RPC 传输——都是基于消息而非基于流的。但因为我们没法省事，消息分帧不得不在 TCP 之上由每个协议用不同的自定义方式重新实现。我现在对“TCP 上做消息分帧”已经肌肉记忆了，各种常见变体都写过。我知道的唯一勉强算例外的是 HTTP/1.1 和 telnet，但 HTTP/1.1 仍是消息导向协议，只是消息大小等于文件；到 HTTP/2 这一点也不成立了。我认为真正的问题是“一切皆文件”这个观念：字节流不是很有用的抽象，“一切皆消息流”会是更好的计算基础隐喻。
josephg查看英文原文
> The data model for TCP is a stream of bytes. However, this is not the right data model for most datacenter applications. Datacenter applications typically exchange discrete messages to implement remote procedure calls This isn't just a datacenter problem. Every single network protocol I've ever created or implemented is message based, not stream based. Every messaging system. Every video game. Every RPC transport. But, because we can't have nice things, message framing has to be re-implemented on top of TCP in a different, custom way by every single protocol. I've basically got message framing-over-TCP in muscle memory at this point, in each of the variants you commonly see. The only kinda-sorta exceptions I know about are HTTP/1.1 and telnet. But HTTP/1.1 is still a message oriented protocol; just with file-sized messages. (And even this stops being true with http2 anyway). In my opinion, the real problem is the idea that "everything is a file". Byte streams aren't a very useful abstraction. "Everything is a stream of messages" would be a far better base metaphor for computing.
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2022Hacker News 讨论帖478 分 · 313 评
同领域还值得读
Paxos vs. Raft：分布式共识是否已达成共识？
Paxos vs. Raft: Have we reached consensus on distributed consensus?
重新审视了 Paxos 与 Raft 两种分布式共识算法之间的差异，试图在一个统一的框架下比较它们，并论证二者的相似程度远高于通常的认知，许多所谓的差别只是表述层面的。
网络与分布式2020 · 343 分 · 65 评 · 引用 10进阶 · 约 28 分钟编辑精选Stellar 协议：互联网级共识的联邦模型
Stellar Protocol: A Federated Model for Internet-Level Consensus (2016)
提出 Stellar Consensus Protocol（SCP），一种面向互联网级共识的联邦拜占庭协议模型。
网络与分布式2018 · 301 分 · 61 评专业 · 约 90 分钟编辑精选拜占庭将军问题（1982）
The Byzantine Generals Problem (1982)
这篇 1982 年论文提出并形式化了拜占庭将军问题：在分布式系统中，部分节点可能叛变或故障、通信不可完全信任时，忠诚节点如何达成一致行动。
网络与分布式2018 · 290 分 · 77 评进阶 · 约 75 分钟编辑精选Google 变差了吗？对搜索引擎中 SEO 垃圾的纵向调查
Is Google Getting Worse? A Longitudinal Investigation of SEO Spam in Search
对搜索引擎结果中的 SEO 垃圾内容做了纵向追踪：作者团队在一段时间内持续采集 Google、Bing、DuckDuckGo 等引擎针对商品评测类查询的结果，用分类器识别联盟营销…
网络与分布式2024 · 276 分 · 274 评入门 · 约 45 分钟编辑精选使 CRDT 能够容忍拜占庭故障
Making CRDTs Byzantine Fault Tolerant
Martin Kleppmann 等人的论文研究如何让 CRDT（无冲突复制数据类型）抵御拜占庭故障节点，即部分参与者恶意篡改数据时仍能保证一致性。
网络与分布式2022 · 266 分 · 40 评进阶 · 约 90 分钟编辑精选