AI 与机器学习2022arxiv.org
为什么基于树的模型在表格数据上仍优于深度学习？
Why do tree-based models still outperform deep learning on tabular data?
进阶 · 约 55 分钟读完前置知识：机器学习基础、梯度提升树（GBDT）、神经网络基础
导读
这篇论文通过大规模基准实验系统比较了树模型（如 XGBoost、LightGBM）与深度学习模型在表格数据上的表现，发现中等规模数据（约 1 万样本）上树模型仍是当前最优，即便不考虑其速度优势。论文进一步从数据特性角度解释差距，例如神经网络对特征的旋转更敏感、更容易受无信息特征干扰，以及目标函数形态的影响。
HN 关注点：大量从业者实践中发现梯度提升树在表格任务上轻易胜过神经网络，讨论焦点在于“1 万样本算不算中等规模”、标题是否夸大，以及树模型自身研究投入为何如此之少。
后来怎么样
该论文成为表格数据学习领域被广泛引用的标准基准，常被用来支撑“树模型仍是表格任务默认首选”这一结论，并推动了对表格数据归纳偏置与专用模型（含表格基础模型）的后续研究。
HN 评论精选
这几条评论里：支持 1 质疑 1 补充 2
支持树模型胜在几乎无需特征工程、天然处理各类数据且易正则化，性价比远高于调教深度模型。
我有个猜想：树模型几乎不需要特征工程。它们能以有原则的方式处理类别特征，对极度偏斜、多峰、异方差的连续数值特征和对 0-1 标准化正态分布一样轻松，而且相比深度模型更容易做正则化（后者可能有数以百万计的参数组合，更别说还要训练到全局最优）。我认为如果你花几个月把数据和模型结构调到最佳，深度模型当然能超过梯度提升树，但何必呢——GBT 今天就能跑完。
CapmCrackaWaka查看英文原文
I have a theory - tree based models require minimal feature engineering. They are capable of handling categorical data in principled ways, they can handle the most skewed/multimodal/heteroskedastic continuous numeric data just as easily as a 0-1 scaled normal distribution, and they are easy to regularize compared to a DL model (which could have untold millions of possible parameter combinations, let alone getting the thing to train to a global optimum). I think if you spent months getting your data and model structure to a good place, you could certainly get a DL model to out-perform a gradient boosted tree. But why do that, when the GBT will be done today?
质疑质疑 1 万样本只能算小数据、标题夸大，并引用附录指出数据量增大时差距会缩小。
> 结果表明，即使不考虑树模型在速度上的优势，它们在中等规模数据（约 1 万样本）上仍是当前最优。 这真的算“中等”吗？在我看来这非常小。MNIST 有 6 万样本，ImageNet 有数百万。我觉得标题夸大了结论。我很好奇这些方法在更大数据集上表现如何，是否存在某个阈值之后深度学习会超过树模型？ 补充：作者在附录里提到了这点——他们把基准扩展到大规模数据集，将训练集截断到 5 万样本（中等规模为 1 万样本），只保留样本数超过 5 万的数据集，这排除了很多数据集使比较不够清晰，但多数情况下增大训练集似乎会缩小神经网络与树模型的差距，严格的研究留待未来工作。
Permit查看英文原文
> Results show that tree-based models remain state-of-the-art on medium-sized data (∼10K samples) even without accounting for their superior speed. Is that really "medium"? That seems very small to me. MNIST has 60,000 samples and ImageNet has millions. I think the title overstates the findings. I'd be interested to hear how these methods compare on much larger datasets. Is there a threshold at which deep learning outperforms tree-based models? Edit: They touch on this in the appendix: > A.2.2 Large-sized datasets > We extend our benchmark to large-scale datasets: in Figures 9, 10, 11 and 12, we compare the results of our models on the same set of datasets, in large-size (train set truncated to 50,000 samples) and medium-size (train set truncated to 10,000 samples) settings. > We only keep datasets with more than 50,000 samples and restrict the train set size to 50,000 samples (vs 10,000 samples for the medium-sized benchmark). Unfortunately, this excludes a lot of datasets, which makes the comparison less clear. However, it seems that, in most cases, increasing the train set size reduces the gap between neural networks and tree-based models. We leave a rigorous study of this trend to future work.
补充批评树模型的正生长停止机制研究投入严重不足，远不如神经网络领域活跃。
让我困惑的是，考虑到树模型如此有效，针对它们的改进研究却少得可怜。比如 LightGBM 和 XGBoost 虽然支持一些正则项，但方差/偏差仍主要靠全局设置最大深度和最大节点数（再靠参数搜索找合适值）来控制。除了数节点数，决定每棵树何时停止生长，肯定存在更强大、更精细的方法吧？如果这是神经网络，早会有上百篇论文提出不同方法并争论优劣了。我不确定原因是神经网络天生更“性感”，还是要在 GBM 上做出 SOTA 改进必须啃硬核的 C++——大概两者都有。
_pastel查看英文原文
It's baffling to me how little research attention there has been to improving tree-based methods, considering their effectiveness. For example, LightGBM and XGBoost allow some regularization terms, but the variance/bias is still mostly controlled by globally setting the max depth and max node count (and then parameter searching to find good settings). Surely there must be more powerful and sophisticated ways of deciding when to stop building each tree than counting the number of nodes? If this was neural nets there would be a hundred competing papers proposing different methods and arguing over their strengths and weaknesses. I'm not sure whether the problem is that neural nets are just fundamentally more sexy, or that in order to make SOTA improvements in GBMs you need to dive into some gnarly C++. Probably both.
补充补充了交互式学习资料，并指出良好正则化的神经网络在表格数据上同样可以表现出色。
如果你对树模型的工作原理感兴趣，我写了决策树的交互式讲解：https://mlu-explain.github.io/decision-tree/ 和随机森林的：https://mlu-explain.github.io/random-forest/ 。另外值得注意，一篇较新的论文表明神经网络只要正则化得当，在表格数据上也能表现良好：https://arxiv.org/abs/2106.11189v1 。
jwilber查看英文原文
If you’re interested in how tree-based models work, I wrote an interactive explanation on decision trees here: https://mlu-explain.github.io/decision-tree/ and random forests here: https://mlu-explain.github.io/random-forest/ It’s also worth noting that a recentish paper shows neural networks can perform well on tabular data if well-regularized: https://arxiv.org/abs/2106.11189v1?utm_source=jesper&utm_medium=email&utm_campaign=coming-up-with-weather-puns-is-a-breeze
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2022Hacker News 讨论帖315 分 · 139 评
- 2024Hacker News 讨论帖212 分 · 111 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选