AI 与机器学习2020arxiv.org
在算法交易问题中使用强化学习
Using Reinforcement Learning in the Algorithmic Trading Problem
进阶 · 约 35 分钟读完前置知识：强化学习基础、A3C 与策略梯度、算法交易与回测基础
导读
论文研究如何用强化学习解决算法交易中的买卖决策问题，评论指出其采用 A3C 并以历史行情回测评估策略。作者报告的回测结果未经实盘验证，因而难以说明策略在真实市场中可盈利。
HN 关注点：强化学习交易论文看似亮眼，但只用历史回测、忽略实盘摩擦与成本，结果是否可信。
HN 评论精选
这几条评论里：质疑 3 亲历 1
质疑只做历史回测、忽略实盘摩擦，且真正盈利者不会公开，因此结果不可信。
我此前写过相关文章，也在高频交易领域短暂工作过，读过几十篇这个主题的论文，可以非常有把握地说，这些结果不可信。这篇论文和该领域几乎所有学术论文一样，最后只是对历史数据做回测，而不是真实系统。回测不仅（非常！）容易过拟合——尤其他们这里用的数据这么少——而且回测与真实世界完全不同。真实世界里有高频交易者抢在你前面、延迟、抖动、手续费、隐藏订单类型、滑点，以及许多一篇 HN 短帖容纳不下的复杂性。每当看到一篇论文以回测收尾，你基本可以认定它是胡扯。这类似于在一个极度简化、没有物理或其他交互的 2D 仿真环境里训练机器人，然后声称造出了真实机器人。很多人犯的错误是相信交易全靠 AI；但现实中，模型往往不如基础设施/延迟/系统/数据问题重要。此外，真正“擅长”交易的人不会发论文，他们默默赚钱。论文通常由从未做出盈利系统的学者或学生发表，他们只是想在简历上添一篇论文。我还没见过一篇好的交易学术论文。
traK6Dcm查看英文原文
As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system. Not only is it (very!) easy to overfit backtests (especially with so little data they are using here), but backtests are nothing like the real world. In the real world there are HFT traders front-running you, latency, jitter, fees, hidden order types, slippage, and a lot of other complexities that don't fit into a short HN post. Whenever you see a paper ending with a backtest you can already assume it's BS. It's similar to training a robot in an extremely simplified 2D simulation environment without physics or other interactions, and then claiming one has built a real robot. A mistake many people make is believing that trading is all about AI. But in reality, the model often matters less than infrastructure/latency/system/data issues. In addition to that, people who are actually "good" at trading don't publish papers, they silently make money. Papers are typically published by academics or students who have never built anything profitable but would like to put a paper on their resume. I have yet to see a single good academic paper about trading. [0] https://www.tradientblog.com/2019/11/lessons-learned-building-an-ml-trading-system-that-turned-5k-into-200k/
亲历交易成败更依赖基础设施、成本和运维；小市场、小样本、短周期、长决策间隔是论文危险信号。
我是前对冲基金和高频交易量化交易员。声称某种策略的论文很多。我不想一上来就愤世嫉俗，但最后还是会走到那一步：交易不只是决定买什么卖什么这个大家觉得性感的部分。我甚至有同事因为离策略更近就觉得自己特殊，结果忽视了不那么光鲜的部分——比如编写读取行情数据、发送订单的软件；比如和经纪商拉关系争取更低成本；还有维护基础设施，不知为何人们觉得这应该算在写代码里。我不是说 RL 帮不上忙，只是把重点放在交易系统的“智能”部分往往会带来失望，因为你会发现模型上有一些你之前没想到的未知限制。比如你发现模型回测做空的那段时间其实禁止做空。我读论文的主要危险信号是：选择小市场的数据集——基本上任何不是美国或西欧大盘股的市场，你会在很晚才发现价格冲击和高额费用；选择市场的一个很小的子集——n 更小，噪声和过拟合更多；时间区间短——还是 n 的问题；决策间隔很长——还是 n 的问题。
lordnacho查看英文原文
Former hedge fund and HFT quant trader here. There's a lot of papers to be found claiming some sort of strategy. I don't want to go to cynicism immediately. But we'll get there: - Trading isn't just about deciding what to buy and sell, the sexy part that everyone thinks is great. I even had colleagues who thought they were special because they worked closer to the strategies, which meant that certain less glamourous parts were neglected. - Less glamourous parts like coding the software to read in the market data and send out orders. - Less glamourous parts like schmoozing with brokers to get them to lower your costs. - And maintaining infrastructure, which somehow people think should come as part of coding. Now I'm not saying that RL won't help you. It's just that focusing on the "intelligent" part of the trading system tends to lead to disappointment, as you discover some unknown restrictions on your model that you hadn't thought of. Things like when you find out short selling was prohibited during the period that your model backtest was shorting. My main red flags when reading papers are: - Choosing a dataset from a small market. Basically any market that isn't the US or Western Europe large caps. You'll discover both price impact and high fees quite late in the game. - Choosing a very small subset of the market. Smaller n, more noise and overfitting. - Short periods. N again. - Long intervals between decision making. N again again. That's not to say there's nothing useful to be read though. You might be inspired by something you come across.
质疑A3C 在对抗性市场容易被对手利用，不适合交易。
还有一个问题是，他们这里用 A3C 来做交易。A3C 已知不适合对抗性环境（例如国际象棋这类棋盘游戏）。我写过一篇论文，证明在棋盘游戏（具体是某些扑克变体）中，A3C 和均匀随机策略一样容易被利用：https://arxiv.org/abs/2004.09677 （Exploitable 是论文中定义的技术术语；大致意思是“一个知道你策略全部信息的人能赢你多少”）。所以如果这个策略在与其他交易者交手后还能存活，我会非常惊讶。
fnbr查看英文原文
An additional problem with this is that they use A3C here for trading. A3C is known to not be suitable for adversarial environments (e.g. board games, like Chess). I wrote a paper that demonstrated that A3C is as exploitable as a uniform random strategy in board games (specifically, some poker variants): https://arxiv.org/abs/2004.09677 (Exploitable is a technical term that is defined in the paper; basically, it's "how much can someone who knows everything about your strategy beat you by?") So I would be very surprised if this survives contact with other traders.
质疑质疑公开动机：真能赚钱就不会公开。
每次这种论文出来，我都忍不住问自己：如果他们真知道这样赚钱，为什么还要告诉任何人？
1024core查看英文原文
Every time such a paper like this comes out, I have to ask myself: if they knew how to make money like this, why would they tell anyone?
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2020Hacker News 讨论帖288 分 · 135 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选