系统与数据库2015vldb.org
Gorilla：一个快速、可扩展的内存时序数据库
Gorilla: A Fast, Scalable, In-Memory Time Series Database (2015)
进阶 · 约 45 分钟读完前置知识：数据库系统基础、压缩编码、分布式存储
导读
这篇论文介绍 Facebook 的 Gorilla 内存时序数据库，目标是支撑大规模运维监控指标的低延迟写入与查询，并降低存储成本。核心做法包括用时间戳 delta 编码和浮点值 XOR 压缩减少内存占用，近期数据放内存、历史数据最终持久化到 HBase 等分布式存储，从而兼顾实时性与水平扩展。
HN 关注点：其时间戳/值压缩技巧和内存+HBase 架构，也围绕“是否真的需要专用时序数据库”展开争论。
后来怎么样
论文提出的时间戳 delta-of-delta 与浮点值 XOR 压缩成为时序数据库的经典编码方案，被后续多款 TSDB 和监控系统借鉴。
HN 评论精选
这几条评论里：质疑 3 补充 1
补充总结论文三项核心技巧：时间戳 delta 编码、值 XOR 压缩、内存缓存+最终持久化 HBase。
几个月前我读了这篇论文并给团队做了总结，这里分享笔记：存储时间序列键时，用第一个时间戳加后续时间戳偏移量（delta）编码能省很多空间；存储值时，连续数据点通常不剧烈波动，比如“每分钟写入数”更可能是 100、99、101，而不是 100、999999、1，因此可以用当前值和前一个值做 XOR 来压缩；作者建议近期数据放内存缓存，最终持久化到 HBase 或类似分布式文件系统，这样既能快速、省空间地拿到实时运维指标，又能为历史存储水平扩展。The Morning Paper 也有一篇不错的分析：https://blog.acolyer.org/2016/05/03/gorilla-a-fast-scalable-in-memory-time-series-database/
pixelmonkey查看英文原文
I read this paper a few months ago and summarized it for my team. Sharing the notes here in case it's helpful for others: - when storing time series keys, you can save a lot of space by encoding them as a first timestamp followed by timestamp offsets (deltas) - when storing time series values, you can save a lot of space by realizing sequential data points tend not to be volatile... e.g. a "writes per minute" series is more likely to be 100, 99, 101, etc. than it is to be 100, 999999, 1, etc. This means you can encode the current value by using XOR tricks with prior values to save space - the authors suggest using in-memory caching for recent data, but doing eventual persistence to HBase or similar distributed filesystems; thanks to this, you can get real-time operational metrics that are fast and space efficient, yet still have a system that scales out horizontally for historical storage The Morning Paper also did a good analysis here: https://blog.acolyer.org/2016/05/03/gorilla-a-fast-scalable-in-memory-time-series-database/
质疑质疑专用 TSDB 的必要性：多数场景 Postgres 足够且更优，所谓特殊性常是误解。
不是要贬低 Gorilla，它看起来很棒，但我不理解大家对专用时序存储的执念。如果你在 Facebook 那种规模上运营，那当然得这么做；但在大多数场景下，你现有的数据库（尤其是 Postgres）是更优选择。时间序列的“特殊”在于大多数人（甚至数学/统计出身的人）对它一无所知，我认为这就是“它需要专用数据库”这一普遍误解的来源。顺便说，我刚写了一篇关于在 PostgreSQL 中存储时序数据的博客：https://grisha.org/blog/2016/12/16/storing-time-series-in-postgresql-part-ii/
gtrubetskoy查看英文原文
Not to bash Gorilla, because it looks like excellent stuff, but I do not understand the general obsession with specialized TS stores. If you're operating on FB-scale, then sure that's what you have to do, but in most cases your database (especially Postgres) is a far superior option. Time series is "unusual" in the sense that most people don't know first thing about it, event folks with degrees in math/statistics. I think this is why there is the prevailing misconception that it requires a specialized database. Incidently I've just written a blog about storing TS in PostgreSQL: https://grisha.org/blog/2016/12/16/storing-time-series-in-postgresql-part-ii/
质疑从运维角度质疑 GlusterFS 细节和 Hadoop/HBase 栈的复杂度，并分享自建类似系统的经验。
1）他们到底用 GlusterFS 做什么、用在哪儿？Gluster 的扩展问题修好了吗，尤其是新存储空间/节点只对新目录和文件可用、对已有目录无效的问题？我上次关注是 2009 年左右，但这是其“无主节点”拓扑的缺陷。2）Facebook 有整支团队维护 Hadoop/HBase，可见这套栈有多庞大。跑过“互联网时间”要求下 Hadoop 的人知道我在说什么：它适合学术/科研里对时间不敏感、延迟计算的任务，很难在按需场景下保持 100% 运行。另外，我无法想象全职只做一个运维产品，那个团队大概会无聊/疲劳。3）希望看到更多网络层信息：传输协议是什么？平均更新的帧大小多大？等等。我们在内部也构建过类似 Gorilla 的系统，很高兴看到结论有重合。
scurvy查看英文原文
A few things here: 1) What/where exactly are they using GlusterFS for? Has Gluster fixed their scaling problems yet? Specifically the issue where new storage spaces/nodes were only available to new directories and files, but not existing directories? Granted, the last time I looked at this was 2009 or so, but it was a flaw due to their "no master node" topology. 2) FB has an entire team to manage Hadoop/HBase. This shows just how much of a beast that stack is. Anyone who has run Hadoop on "Internet time" knows what I'm talking about. It's great at running time insensitive, deferred compute jobs in an academic or scientific setting. It's really hard to keep it all 100% running in an on-demand setting. Aside, I couldn't imagine just working on 1 product in an operations setting as my full-time job. Boredom/fatigue must be a problem on that team. 3) I'd like to see more information on the networking side. What transport protocol? How large are the average updates in frame size? Etc etc. We've built something similar to Gorilla in-house, so I'm happy to see that we've come to some of the same conclusions.
质疑遗憾缺少与金融业标准 KDB 的基准对比，并指出开源方案仍难企及。
我很希望这篇论文能和 KDB 做个对比。KDB 许可证不便宜，他们肯定不会为了让你发表针对它的基准测试就给测试许可证，但在金融业它就是 TSDB 的标准。开源方案里从来没有接近它的。
saosebastiao查看英文原文
I really wish this included a comparison with KDB. It's not cheap to get a license, and they certainly wouldn't give a testing license in order to publish benchmarks against it, but in finance it is the standard for TSDBs. There hasn't ever been anything open source that has come close.
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2015Hacker News 讨论帖35 分 · 20 评
- 2016Hacker News 讨论帖125 分 · 56 评
同领域还值得读
I/O 比 CPU 更快——让我们划分资源并消除操作系统抽象
I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions
指出在现代服务器硬件上 I/O 速度已接近甚至超过 CPU，传统操作系统抽象反而成为瓶颈。
系统与数据库2019 · 817 分 · 277 评进阶 · 约 30 分钟编辑精选使用 io_uring 的高效 I/O
Efficient IO with io_uring
由 Linux 内核开发者 Jens Axboe 撰写，介绍 Linux 新的异步 I/O 接口 io_uring。
系统与数据库2019 · 426 分 · 96 评进阶 · 约 45 分钟编辑精选线程对大多数用途而言是个坏主意（1995）
Threads Are a Bad Idea for Most Purposes (1995)
这篇 1995 年的文章由 John Ousterhout 撰写，讨论为什么线程对大多数用途而言是坏主意，并主张事件驱动编程通常更简单、更高效。
系统与数据库2017 · 423 分 · 382 评进阶 · 约 25 分钟编辑精选学习型索引结构之论证
The Case for Learned Index Structures
提出把索引看作一种"模型"：既然 B-tree 等结构本质上是在学习数据的分布，就可以用机器学习模型（如回归模型、神经网络）来近似键到位置的映射，从而替代 B-tree、哈希索引…
系统与数据库2017 · 398 分 · 66 评 · 引用 25进阶 · 约 60 分钟编辑精选FoundationDB：一个分布式、解耦的事务型键值存储
FoundationDB: A distributed, unbundled, transactional key value store
介绍 FoundationDB：一个分布式、解耦设计的事务性键值存储。
系统与数据库2021 · 283 分 · 99 评进阶 · 约 90 分钟编辑精选