编程语言与软件工程2020plateau-workshop.org
DataFrame 只是表吗？（2019）
Is a Dataframe Just a Table? (2019)
入门 · 约 30 分钟读完前置知识：pandas 或 R 基本使用、SQL 基础
导读
这篇论文（PLATEAU 2019 短文）探讨“DataFrame 是否只是一张表”这一问题，比较了 pandas、R、SQL 等系统中 DataFrame 的语义差异，指出 DataFrame 概念缺乏统一定义、API 繁多且行为不一致，给使用者带来理解与迁移上的困难。
HN 关注点：HN 读者长期被 pandas 等 DataFrame API 的不一致和难用困扰，讨论聚焦于 DataFrame 与关系模型/表的本质区别，以及是否该回归 SQL 或设计更好的语言。
HN 评论精选
这几条评论里：支持 2 质疑 1 补充 1
支持认同论文对 DataFrame API 繁杂不一致的批评，并推荐 R 的 data.table 作为更高效的替代。
论文提到：同一逻辑有多种表达方式，使异构风格的程序难以理解；此外超过 200 个重载 API 且默认参数随版本变化，很难记住。我对此深有同感——我一直觉得 pandas 用起来别扭，不确定自己写法是否高效、地道，代码风格难以保持一致。后来用 R（尤其 data.table 包）效率高了很多。
leto_ii查看英文原文
> Having many different ways to express the same logic makes it hard for developers to understand programs of heterogeneous styles. Besides having varying ways to express the same simple logic, the sheer number of APIs (> 200) that are not only overloaded but also have default parameters that may change version to version, making it hard to remember the APIs. It's a bit tangential to the main point, but I do agree with this remark. I have always found Pandas uncomfortable to work with. I'm never sure if I'm doing things in the most efficient/idiomatic way and I've found it hard to be consistent over time, especially since I've picked up different bits of code from different places. I've gotten a lot more efficiency out of R, especially the data.table package.
补充认为 DataFrame 之争偏离了根本：核心是关系模型与关系代数，SQL 只缺长宽表转换等少数运算。
表和 DataFrame 都只是更根本的东西——数据的关系模型——的末端产物。范式太简单直观，以至于分不清人们是有意围绕它设计还是碰巧走对了路。我认为争论表与 DataFrame 的区别（以及查询语言之争）是舍本逐末；真正重要的是关系模型和支持关系代数的系统。SQL 唯一的重大缺陷是缺少类似 R tidyverse 的 gather()/spread() 这类运算。
roenxi查看英文原文
Tables and data frames are both leaves in the far more fundamental flow that beginners don't pick up because it is too simple - the relational model of data. The real problem is the the basic normal forms are so obvious and simple it is difficult to tell if people designed around them on purpose or stumbled onto the right path. I suspect the distinction between tables and data frames (and arguing about query languages) is from people who have lost sight of the important things when dealing with data - the relational model and a system that supports relational algebra. Beyond that, features and optimisations need to be justified in terms of today's (and tomorrow's) needs which are specific to the data. The only major issue with SQL is a lack of support for something like R's tidyverse gather() and spread() operations which could be considered as missing operations in relational algebra.
支持批评 DataFrame 语义边界模糊、行为不可预测，宁可使用语义明确的 SQL。
DataFrame 比表更糟。它没有清晰的能力边界，你甚至可以把控制流逻辑塞进去，使它变成一个用 DSL 描述的任意计算图的执行器，行为完全取决于开发者的实现。我宁愿用 SQL，因为至少能明白它在做什么。
tanilama查看英文原文
It is worse than a table. Dataframe don't really have a clear boundary as what it can do or cant do. I see no problem why you can't shoehorn some control flow logic into it which makes it pretty much just an executor of arbitrary computation graph specified in DSL, which relies on however the developer decides to implement it. I'd rather take SQL because I have a better understanding what it is doing.
质疑认为 DataFrame 本质就是表，但分析场景可简化类型系统；建议用函数式语言统一表处理并解决 SQL 难组合的问题。
DataFrame 基本上就是一张表（GraphQL 也只是对 SQL 的糟糕再发明）。不过要公平地说：数据库需要关心存储约束和外键（所以有很多列类型），而纯分析场景（如 pandas）基本只需要数字和字符串两种类型（SAS 在这点上做对了）。出路或许是设计一种类 Haskell 的函数式表处理语言（不必图灵完备），兼具 SQL 的声明性、可组合、可编译到不同后端——LINQ 就是类似思路。
js8查看英文原文
> What should we make of it? People are idiots! (I am sorry to say that, I don't really mean it, I empathize, everyone sometimes is.) Yes, dataframe is pretty much just a table. (And yes, GraphQL is a poor reinvention of SQL.) However, to be fair, there are different considerations. Database needs to know things like storage constraints and foreign keys (so you have many different column types), when you're doing just analytics (i.e. pandas), you pretty much only need two types - number and (trimmed) string (and sometimes a datetime, but that's just conveniently formatted number). (I think SAS got that right.) Anyway, I think the way out of this mess is to have a functional, Haskell-like language for processing tables (not necessarily Turing complete) that would subsume declarativness of SQL, and that could be compiled into different targets. The language would basically specify some (limited, so recursion in that language would not be allowed) processing of tables (of known types), and you could then compose these processing pieces, or integrate them with custom processing in other language. I understand why people hate SQL, it is on some level hard to write and compose. I think a correctly designed functional language would help here. Or maybe just tear LINQ out of the .NET ecosystem.
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2020Hacker News 讨论帖193 分 · 113 评
同领域还值得读
Google 的软件工程（2017）
Software Engineering at Google (2017)
介绍了 Google 的软件工程实践，包括其单一代码库（monorepo）、几乎全部开发在主干进行而非分支、代码审查、构建系统 Blaze、协议缓冲区等工具与流程，并讨论了频繁重…
编程语言与软件工程2017 · 1,127 分 · 465 评 · 引用 15入门 · 约 45 分钟编辑精选Peter Naur：编程作为理论构建 (1985)
Peter Naur – Programming as Theory Building (1985)
Peter Naur 1985 年的经典论文《Programming as Theory Building》提出：编程的本质不是写代码或文档，而是在程序员头脑中构建对程序及其运行…
编程语言与软件工程2016 · 629 分 · 74 评入门 · 约 45 分钟编辑精选SciPy 1.0：Python 科学计算的基础算法
SciPy 1.0: fundamental algorithms for scientific computing in Python
这篇发表于 Nature Methods 的论文介绍了 SciPy 1.0——Python 科学计算生态的基石库，涵盖其核心算法模块（优化、积分、插值、线性代数等）、软件架构与社…
编程语言与软件工程2020 · 387 分 · 98 评入门 · 约 45 分钟编辑精选Lisp 的历史（1979）
History of Lisp (1979)
这是 Lisp 发明者 John McCarthy 于 1979 年为 HOPL（编程语言史）会议撰写的回顾文章，讲述 Lisp 的起源、设计动机与早期演进，文中也涉及 if 表…
编程语言与软件工程2018 · 352 分 · 44 评入门 · 约 45 分钟编辑精选使用 NumPy 进行数组编程
Array Programming with NumPy
这篇 Nature 文章介绍 NumPy 的数组编程模型及其在科学计算中的核心作用：以 ndarray 为中心，通过向量化操作让 Python 高效处理多维数组。
编程语言与软件工程2020 · 318 分 · 134 评入门 · 约 40 分钟编辑精选