编程语言与软件工程2018conal.net编辑精选
自动微分的简单本质
The Simple Essence of Automatic Differentiation
专业 · 约 120 分钟读完前置知识：范畴论基础、函数式编程、自动微分基础
导读
Conal Elliott 的这篇 ICFP 2018 最佳论文提出了一种自动微分的范畴论表述：将程序转换为一阶/范畴表示后，反向模式 AD 可以通过 CPS（续体传递风格）变换加上与正向模式相同的简单求导规则来实现。作者声称由此得到的算法不需要计算图、tape、变量或 mutation，天然可并行且构造上正确。
HN 关注点：是因为 AD 是 PyTorch/TensorFlow 等深度学习框架的核心机制，而这篇论文声称用更简洁的理论框架统一正反向模式并避免图/tape 的实现复杂度，引发了对其实用性和性能的讨论。
HN 评论精选
这几条评论里：支持 1 质疑 1 补充 2
质疑质疑论文“无图、并行友好”的声明，指出缺少性能基准数据
这里有更多资料：https://github.com/conal/talk-2018-essence-of-ad。不过有人读过论文吗？我搞不懂论文声称的“与常用 RAD 实现不同，这里的算法不涉及图、tape、变量、偏导数或 mutation，天然并行友好、构造上正确”。这听起来很棒，但并不显然。反向模式通常极不并行友好，因为本质上要遍历计算图，而图的划分开销可能比串行遍历还高。论文说不建图、让函数和导数一起传递，这没问题，但这种方法真的能产生更并行友好的代码吗？效率能否与传统图/tape 方法相当？我没看到任何 benchmark，真正有用的是运行时间与加 AD 后计算梯度的时间之比，以及该比值随规模扩大如何变化。
kxyvr查看英文原文
There's more information here: https://github.com/conal/talk-2018-essence-of-ad That said, has anyone had a look at the paper? What I can't figure out is the assertion that, "In contrast to commonly used RAD implementations, the algorithms defined here involve no graphs, tapes, variables, partial derivatives, or mutation. They are inherently parallel-friendly, correct by construction, and usable directly from an existing programming language with no need for new data types or programming style, thanks to use of an AD-agnostic compiler plugin." That would be fantastic, but that's not immediately obvious to me. Generally, reverse mode is extremely non-parallel friendly because we're essentially traversing a computation graph. That traversal can be done in parallel, but figuring out how to partition that graph is probably more expensive that traversing it in serial. Now, the paper makes a big deal about not building that graph and passing along the derivative at the same time as the function. Fine. But does that method really produce a more parallel friendly code and is that as efficient as a more traditional graph or tape based method? Unless I'm missing something, I don't see any benchmarks of the method and what would be really helpful would be a ratio of the run time of a routine versus the run time of the routine with the AD method instrumented and the gradient calculated. The big question is how does this ratio hold as the method is scaled up in terms of variables. If the parallel claim holds true, then we should be able to also calculate the resulting parallel efficiencies. Anyway, I'm genuinely curious if someone has additional information about the computational viability.
补充补充资料：论文获奖，并附作者在 MSR 的演讲视频
这篇论文获得了 ICFP 2018 最佳论文奖。Conal Elliott 最近在微软研究院做了一个更长的报告：https://www.youtube.com/watch?v=ne99laPUxN4
bmc7505查看英文原文
This paper won a best paper award at ICFP 2018. Conal Elliott recently gave a longer talk about it at Microsoft Research: https://www.youtube.com/watch?v=ne99laPUxN4
支持专业解读：反向模式 AD = CPS 变换 + 正向模式同样的求导规则
比其他评论说得更清楚些，这篇论文的核心洞见是：在把程序变换到一阶/范畴表示的前提下，反向模式 AD 其实就是 1) 把程序变换到续体传递风格的范畴（这会“翻转”程序范畴中复合操作的形状）；2) 做与正向模式相同的计算/变换，也就是高中就学的那种简单求导。再加一个小技巧就能更好，但这就是核心。也就是说反向模式的实现可以（几乎）和正向模式一样简单——前提是语言处于这种范畴化的一阶表示。能把反向模式讲得如此简单，本身就是巨大的创新。
carterschonwald查看英文原文
To state it more clearly than some of the other comments, part of the code insight this paper provides is the following: Subject to transformation of your program into the first order / categorical representation, reverse mode auto differentiation is simply about 1) transforming the program into the continuation passing style category (this “flips” the shape of the compose operation in the category of our program) 2) do the same calculation / transform as you’d do for forward mode. Aka the really simple derivative calc we learn in high school. There’s an extra trick to be done to make things even better, but that’s the meat of it. That reverse mode becomes (almost) as simple as forward mode to implement. Subject to your language being in this sort of categorical first order representation :) Reverse mode being simple by any name to explain is a huge innovation alone.
补充提问：这些理论对只使用框架的普通 ML 工程师是否有实用价值
标题看起来很酷！有人能告诉我，对我这样的普通 ML 工程师有什么学习动机吗？这些想法似乎对开发 PyTorch 这类深度学习框架的人很有帮助，但对只用框架的人呢？不过无论对我有没有用，我都很尊重能发表优秀论文的研究者。
warabe查看英文原文
The title seems pretty cool! Could anyone care to tell me if there is a motivation to learn this topic for ordinary ML enginneer such as myself. It seems these ideas presented in this paper are really helpful for those who develop DL frameworks like pytorch, but what about for those who only use frameworks? However, regardless of how useful these ideas for me, I really respect researchers who publish excellent papers.
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2018Hacker News 讨论帖167 分 · 16 评
同领域还值得读
Google 的软件工程（2017）
Software Engineering at Google (2017)
介绍了 Google 的软件工程实践，包括其单一代码库（monorepo）、几乎全部开发在主干进行而非分支、代码审查、构建系统 Blaze、协议缓冲区等工具与流程，并讨论了频繁重…
编程语言与软件工程2017 · 1,127 分 · 465 评 · 引用 15入门 · 约 45 分钟编辑精选Peter Naur：编程作为理论构建 (1985)
Peter Naur – Programming as Theory Building (1985)
Peter Naur 1985 年的经典论文《Programming as Theory Building》提出：编程的本质不是写代码或文档，而是在程序员头脑中构建对程序及其运行…
编程语言与软件工程2016 · 629 分 · 74 评入门 · 约 45 分钟编辑精选SciPy 1.0：Python 科学计算的基础算法
SciPy 1.0: fundamental algorithms for scientific computing in Python
这篇发表于 Nature Methods 的论文介绍了 SciPy 1.0——Python 科学计算生态的基石库，涵盖其核心算法模块（优化、积分、插值、线性代数等）、软件架构与社…
编程语言与软件工程2020 · 387 分 · 98 评入门 · 约 45 分钟编辑精选Lisp 的历史（1979）
History of Lisp (1979)
这是 Lisp 发明者 John McCarthy 于 1979 年为 HOPL（编程语言史）会议撰写的回顾文章，讲述 Lisp 的起源、设计动机与早期演进，文中也涉及 if 表…
编程语言与软件工程2018 · 352 分 · 44 评入门 · 约 45 分钟编辑精选使用 NumPy 进行数组编程
Array Programming with NumPy
这篇 Nature 文章介绍 NumPy 的数组编程模型及其在科学计算中的核心作用：以 ndarray 为中心，通过向量化操作让 Python 高效处理多维数组。
编程语言与软件工程2020 · 318 分 · 134 评入门 · 约 40 分钟编辑精选