系统与数据库2025db.cs.cmu.edu
F3：面向未来的开源数据文件格式
F3: Open-source data file format for the future
进阶 · 约 60 分钟读完前置知识：列式存储基础、Parquet 格式、数据序列化（如 Flatbuffers）
导读
CMU 等团队提出的 F3 是一个开源数据文件格式，旨在取代 Parquet 等现有列式格式。它采用 Flatbuffers 存放元数据（可零拷贝直接解析）、更强大的可组合轻量级编码，并去掉了 Parquet 复杂的 Dremel 记录切分设计，解决了元数据解析开销和安全性等老问题。
HN 关注点：能否成为下一代开放数据格式：Parquet 的种种缺陷长期被数据库社区诟病，而作者阵容（含 Pandas 创始人 Wes McKinney）和吸收 BtrBlocks/FastLanes 等新编码思路让人期待。
HN 评论精选
这几条评论里：支持 1 质疑 1 补充 2
支持逐条对比 Parquet 的缺陷（元数据安全、解析开销、编码弱、Dremel 复杂度、行定位），认为 F3 都有针对性改进
粗看了一下，它解决了很多 Parquet 的短板，令人兴奋：1) Parquet 元数据用 Thrift，注释里写着“若此字段存在则另一字段必须存在”，但没有任何代码真正校验，喂入伪造元数据很可能让读取器崩溃；2) Parquet 元数据必须解析出来，需要分配缓冲区、读取字节、再动态分配一堆东西，堆分配太多；F3 的 Flatbuffers 方案可以直接解释字节，解决了这个问题；3) 编码更强大，数据库社区呼吁可组合/递归的轻量编码很久了，BtrBlocks 和 FastLanes 都比 Parquet 好得多，很高兴这些想法被吸收；4) Parquet 的 Dremel 记录切分让人头大，毫无实际好处却把格式复杂化，很高兴被去掉了；5) Parquet 数据页行数不定，找某一行得扫描整个 ColumnChunk，而 F3 似乎可以直接定位。
sakras查看英文原文
Giving it a quick look, seems like they've addressed a lot of the shortcomings of Parquet which is very exciting. In no particular order: - Parquet metadata is Thrift, but with comments saying "if this field exists, this other field must exist", and no code actually verifying the fact, so I'm pretty sure you could feed it bogus Thrift metadata and crash the reader. - Parquet metadata must be parsed out, meaning you have to: allocate a buffer, read the metadata bytes, and then dynamically keep allocating a whole bunch of stuff as you parse the metadata bytes, since you don't know the size of the materialized metadata! Too many heap allocations! This file format's Flatbuffers approach seems to solve this as you can interpret Flatbuffer bytes directly. - The encodings are much more powerful. I think a lot of people in the database community have been saying that we need composable/recursive lightweight encodings for a long time. BtrBlocks was the first such format that was open in my memory, and then FastLanes followed up. Both of these were much better than Parquet by itself, so I'm glad ideas from those two formats are being taken up. - Parquet did the Dremel record-shredding thing which just made my brain explode and I'm glad they got rid of it. It seemed to needlessly complicate the format with no real benefit. - Parquet datapages might contain different numbers of rows, so you have to scan the whole ColumnChunk to find the row you want. Here it seems like you can just jump to the DataPage (IOUnit) you want. - They got rid of the heavyweight compression and just stuck with the Delta/Dictionary/RLE stuff. Heavyweight compression never did anything anyway, and was super annoying to implement, and basically required you to pull in 20 dependencies. Overall great improvement, I'm looking forward to this taking over the data analytics space.
补充作者背书：Pandas 之父参与让格式可信且易获生态支持
看到 Wes McKinney 的名字我就信了。他是 Python 表格分析标配库 Pandas 的创造者，这让该格式从一开始就获得广泛支持，而且他几十年持续关注这个问题，见解格外有价值。
anigbrowl查看英文原文
> Wes McKinney Sold For those who don't know. Wes McKinney is the creator of Pandas, the go-to tabular analysis library for Python. That gives his format widespread buy-in from the outset, as well as a couple of decades' of Caring About The Problem which makes his insights unusually valuable.
质疑质疑：需要专用解码器的格式违背了开放格式的初衷
一种需要专门程序才能解码的格式太离谱了，这相当于每个 zip 文件都捆绑一个 7zip。
fallat查看英文原文
A format requiring a program to decode is nuts. Might as well bundle 7zip with every zip file.
补充外行提问：想理解列式存储的意义和这项工作的实际价值
恕我不太懂列式存储的区别，为什么这很酷？大赢家是不是可以随身发送内置沙箱的自定义向量嵌入小数据库？我从论文里感受到“奠基性构件”的气质，可惜每页只能看懂几句，不过图片很漂亮，配色大胆而有品味，容易被说服的我竖起两个大拇指。
digdugdirk查看英文原文
Forgive me for not understanding the difference between columnar storage and otherwise, but why is this so cool? Is the big win that you can send around custom little vector embedding databases with a built in sandbox? I get the "step change/foundational building block" vibe from the paper - and the project name itself implies more than a little "je ne sais quoi" - but unfortunately I only understand a few sentences per page. The pictures are pretty though, and the colour choices are tasteful yet bold. Two thumbs up from the easily swayed.
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2025Hacker News 讨论帖386 分 · 125 评
同领域还值得读
I/O 比 CPU 更快——让我们划分资源并消除操作系统抽象
I/O Is Faster Than CPU – Let’s Partition Resources and Eliminate OS Abstractions
指出在现代服务器硬件上 I/O 速度已接近甚至超过 CPU，传统操作系统抽象反而成为瓶颈。
系统与数据库2019 · 817 分 · 277 评进阶 · 约 30 分钟编辑精选使用 io_uring 的高效 I/O
Efficient IO with io_uring
由 Linux 内核开发者 Jens Axboe 撰写，介绍 Linux 新的异步 I/O 接口 io_uring。
系统与数据库2019 · 426 分 · 96 评进阶 · 约 45 分钟编辑精选线程对大多数用途而言是个坏主意（1995）
Threads Are a Bad Idea for Most Purposes (1995)
这篇 1995 年的文章由 John Ousterhout 撰写，讨论为什么线程对大多数用途而言是坏主意，并主张事件驱动编程通常更简单、更高效。
系统与数据库2017 · 423 分 · 382 评进阶 · 约 25 分钟编辑精选学习型索引结构之论证
The Case for Learned Index Structures
提出把索引看作一种"模型"：既然 B-tree 等结构本质上是在学习数据的分布，就可以用机器学习模型（如回归模型、神经网络）来近似键到位置的映射，从而替代 B-tree、哈希索引…
系统与数据库2017 · 398 分 · 66 评 · 引用 25进阶 · 约 60 分钟编辑精选FoundationDB：一个分布式、解耦的事务型键值存储
FoundationDB: A distributed, unbundled, transactional key value store
介绍 FoundationDB：一个分布式、解耦设计的事务性键值存储。
系统与数据库2021 · 283 分 · 99 评进阶 · 约 90 分钟编辑精选