经济与社会科学2018papers.ssrn.com
A/B 测试中的 p-hacking 与假阳性发现
p-Hacking and False Discovery in A/B Testing
入门 · 约 45 分钟读完前置知识：假设检验基础、p 值与假阳性概念、A/B 测试基本流程
导读
这篇论文研究 A/B 测试中的 p-hacking（操纵 p 值）与假发现率问题。作者基于 Optimizely 2014 年的数据分析发现，传统 t 检验方式下大量实验结果存在 p-hacking，假阳性比例相当高；Optimizely 于 2015 年引入了新的统计方法来缓解这一问题。
HN 关注点：HN 读者多为工程师和产品从业者，A/B 测试是日常工作，讨论焦点是实践中普遍存在的统计误用——尤其是“偷看结果提前停止实验”导致的假阳性。
后来怎么样
这篇论文推动了 A/B 测试平台统计方法的改进，Optimizely 于 2015 年引入序列检验等新统计引擎以抑制 p-hacking，成为业界实验平台方法演进的代表性工作。
HN 评论精选
这几条评论里：补充 2 亲历 2
亲历以亲历说明科技公司实验规范缺失，p-hacking 比论文估计的更普遍
如果假发现率只有 57% 我都会震惊。实习时我发现负责 A/B 测试的分析师往往没有科学或实验背景，也不在乎。我们团队要求预注册假设、按预定时长跑实验，结果被开发者嫌“太较真”。科技行业的环境并不利于做实验：节奏太快，等你搞清某个结果是胡扯时，当事人已经升了三次职、跳去别家公司当总监了。我现在转行做科研，虽然大家照样疯狂 p-hack，但至少还有些羞耻感，而且有长期代价——我见过有研究者花多年试图复现自己早年用可疑手段得出的发现。
boron1006查看英文原文
I would be shocked if it were as low as 57%. As an intern, I found that the analysts in charge of A/B tests often didn't have a background in science or running experiments, and didn't really care. There were a couple of data analytics teams in the company, and I think a lot of the developers didn't like my team because we were seen as "fussier" than the other one. We required people to preregister hypotheses, and run experiments for predetermined amounts of time. I don't think the tech environment is very conducive to running experiments. Everything moves too fast, by the time you figure out the results someone gives you are bs, they've already got promoted 3 times and work as a director at a different company. I work in science now, and although people still p-hack like hell, there's at least some sort of shame about it. There's a long term cost too, I've met a couple researchers who have spent years trying to replicate some finding they got early in their career through suspicious means.
补充作者澄清数据来源并回应质疑
我是论文作者之一。我们发现人们用传统 t 检验时会 p-hack，过去大多数 A/B 测试都是这么做的。论文用的是 Optimizely 2014 年的数据，早于其 2015 年推出的旨在解决这些问题的测试方法。想了解 Optimizely 如何防止 p-hacking，可以看其当前统计引擎的白皮书。
aisscott查看英文原文
Hi, I am one of the authors. We found that people p-hack with traditional t-tests. Most A/B tests were run this way in the past and some still are. The paper is using Optimizely data (from 2014) before Optimizely introduced new testing in 2015 designed to solve the issues we found in the paper. If you want to know how Optimizely prevents p-hacking check out the math behind Optimizely’s current testing here: https://www.optimizely.com/resources/stats-engine-whitepaper/
亲历亲历案例：从业者普遍“跑到显著为止”，缺乏统计常识
在一次编程大会上，我和一位知名公司的资深开发者聊天，他滔滔不绝讲他们的 A/B 测试。我问他们怎么决定实验跑多久，答案是“直到出现显著结果”。我当时很震惊，现在对这类回答已经习以为常了……还有开发者认为假阳性根本不存在。
nanis查看英文原文
Once at a programming conference, I was talking with a very senior developer at a well known company. He was going on and on about their A/B testing efforts. I asked how they decided how long they would run an experiment for. The answer was "until we get a significant result." I was shocked then, but now I am used to getting these kinds of responses from developers ... That and a belief that false positives are not a thing.
补充从业者补充：贝叶斯方法是解决 peeking 问题的一种方案
我是 VWO 的创始人。我们 2015 年把测试引擎改成了贝叶斯方法，以解决频率学派方法中的“偷看问题”。可以读我们关于 SmartStats 的博客了解具体做法。
paraschopra查看英文原文
Hi, founder of VWO here. We revamped our testing engine to Bayesian in 2015 to prevent the ‘peeking problem’ with frequentists approaches. You can read about our approach https://vwo.com/blog/smartstats-testing-for-truth/
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2018Hacker News 讨论帖299 分 · 104 评
同领域还值得读
一切的回报率，1870–2015
The Rate of Return on Everything, 1870–2015
利用1870–2015年多国的长期数据，估算股票、住房等各类资本的回报率，并将其与经济增长率进行比较。
经济与社会科学2019 · 487 分 · 260 评进阶 · 约 100 分钟编辑精选一切资产的回报率，1870–2015
The Rate of Return on Everything, 1870–2015
构建了 1870–2015 年 16 个国家的长周期资产回报数据集，覆盖股票、住房、政府债券和短期票据，系统比较各类资产的真实回报与波动性。
经济与社会科学2018 · 215 分 · 151 评进阶 · 约 120 分钟编辑精选一切资产的回报率，1870–2015（2019）
The Rate of Return on Everything, 1870–2015 (2019)
这篇发表于《经济学季刊》的论文构建了 1870–2015 年间 16 个主要发达国家的股票、债券、房地产等资产收益率的长周期数据库。
经济与社会科学2024 · 190 分 · 148 评进阶 · 约 80 分钟编辑精选想法是否越来越难找？
Are ideas getting harder to find?
探讨“想法是否越来越难找到”：通过衡量多个领域的“研究生产率”，发现维持相同的增长需要投入指数级增长的研究资源，即研究效率在持续下降。
经济与社会科学2016 · 182 分 · 106 评入门 · 约 60 分钟编辑精选比特币与区块链的经济极限
The Economic Limits of Bitcoin and the Blockchain
芝加哥大学 Eric Budish 的论文从经济学角度分析比特币与区块链的根本限制：核心论点是只要攻击网络（双花或砸盘做空）的收益超过挖矿成本，51% 算力攻击就有利可图，而挖矿…
经济与社会科学2018 · 176 分 · 37 评进阶 · 约 90 分钟编辑精选