AI 与机器学习2020arxiv.org编辑精选
GPT-3：语言模型是少样本学习者
GPT-3: Language Models Are Few-Shot Learners
入门 · 约 90 分钟读完前置知识：Transformer 基础、语言模型预训练、NLP 基准评测
导读
OpenAI 的 GPT-3 论文训练了一个 1750 亿参数的超大规模语言模型，仅通过自回归预训练（无微调），就能在翻译、问答、填空、算术等众多 NLP 任务上以零样本或少样本（few-shot）方式取得有竞争力的结果。论文展示了大规模语言模型作为“少样本学习器”的潜力，并讨论了相关的偏见与伦理问题。
HN 关注点：GPT-3 的规模（比此前最大的 T5 大一个数量级以上）和无需微调的少样本能力引发了对大模型能力边界、缩放定律和标题修辞的广泛讨论。
后来怎么样
GPT-3 的发布引发了大规模语言模型研究浪潮，few-shot/zero-shot prompting 成为标准范式，直接催生了 InstructGPT、ChatGPT 等后续工作，并推动了提示工程这一新领域的形成。
HN 评论精选
这几条评论里：支持 1 质疑 1 补充 2
支持从规模、无微调、少样本三方面论证 GPT-3 是重大突破
这在我看来是件大事：1) 作者成功训练了 1730 亿参数的模型，此前文献中最大的 Google T5 只有 110 亿参数；GPT-3 的权重仅用 Float32 表示就占约 700GB 内存。虽然比人脑数百万亿突触还小三个数量级，但万亿参数级别的模型突然看起来是可行的了。2) 模型在许多 NLP 任务和基准上无需微调就取得有竞争力的结果——只有无监督（自回归）预训练，对每个下游任务只给文本指令和可能的问答示例，涵盖翻译、问答、完形填空、词语重组、新词造句和三位数算术。3) 模型只在零样本或少样本（通常 10 到 100 个示例）设置下测试，就像人类一样不需要大量示例就能在新任务上表现不错。
cs702查看英文原文
This looks like a big deal to me: 1. First of all, the authors successfully trained a model with 173 BILLION PARAMETERS. The previous largest model in the literature, Google’s T5, had "only" 11 billion. With Float32 representations, GPT-3-173B's weights alone occupy ~700GB of memory (173 billion params × 4 bytes/param). A figure in the 100's of billions is still 3 orders of magnitude smaller than the 100’s of trillions of synapses in the human brain [a], but consider this: Models with trillions of weights are suddenly looking... achievable. 2. The model achieves competitive results on many NLP tasks and benchmarks WITHOUT FINETUNING. Let me repeat that: there is no finetuning. There is only unsupervised (i.e., autoregressive) pretraining. For each downstream NLP task or benchmark, the pretrained model is given text instructions, and possibly sample text with questions and answers. The NLP tasks on which the model was tested include translation, question-answering, cloze tasks, unscrambling words, using novel words in sentences, and performing 3-digit arithmetic. 3. The model is tested only in a ZERO-SHOT or FEW-SHOT setting. In other words, for each NLP task, the pretrained model is given text instructions with zero examples, or text instructions with a small number of examples (typically 10 to 100). As with human beings, GPT-3-173B doesn't need lots of examples to perform competitively in novel NLP tasks. 4. The results reported by this paper on all NLP tasks and benchmarks should be seen as a BASELINE. These results likely could be meaningfully improved with conventional finetuning. 5. The model’s text generation FOOLS HUMAN BEINGS, without having to cherry-pick examples. -- [a] https://www.google.com/search?q=number+of+synapses+in+human+brain
补充总结 GPT-2 到 GPT-3 的范式变化，并指出大模型并非全面碾压小模型
通读了论文大部分内容。GPT-2 时代想生成诗歌需要大量诗歌来微调才能得到合理结果；GPT-3 则改用少样本学习，无需对每个样本做梯度更新。论文很长，大量篇幅是与 Grover、T5 等模型的对比，表现不错——但这是 1750 亿参数的模型（对比 Grover/T5 的 15 亿/110 亿），说明在某些场景下小参数模型仍能胜过大模型。另外我认为论文对这类模型的伦理、道德和偏见问题解释得很好。
sdan查看英文原文
Read though most of the paper and here's what GPT-3 is: If you wanted to generate poems with GPT-2, you'd need to have a lot of poems to fine-tune GPT-2 to get reasonable results. With GPT-3, you use few-shot learning instead (without the need to do gradient updates with each example) The paper is long and filled with how it stacks with models like Grover and T5 and it does well... given that this is a 175 B param model (relative to Grover/T5's 1.5/11B param models). This shows that even with these huge models, smaller models can outperform them in certain instances with lesser param models. Also I think they did a good job with explaning the ethics and morals around what models like these mean / what biases this has.
质疑批评论文标题的修辞暗示了未经论证的“元学习”主张
我不喜欢 OpenAI 近来“语言模型是 X”这种标题趋势，我认为这是一种损害讨论质量的修辞手法。同一篇论文完全可以叫《用大规模语言模型做少样本学习》。但似乎存在一种想在神经网络中看到某种不可言说特质的强烈愿望——它就是个语言模型，做语言建模，恰好能用于少样本学习且效果惊人。除此之外，说它“是”少样本学习者意味着什么？严格意义上是同一个主张，但它暗示了更宏大的论断：语言建模/无监督学习作为长上下文任务本身就蕴含元学习能力——这个命题很难严谨表述，更难论证，而标题正是把这种论证偷偷塞了进来。别误会：他们的直觉合理，结果令人印象深刻（尽管有明显差距），但作为备受公众关注的实验室，他们有责任更谨慎地表述结果，而我认为他们忽视了这一点。
julianjm查看英文原文
I am not a fan of this trend of "Language Models Are X" in recent work particularly out of OpenAI. I think it's a rhetorical sleight of hand which hurts the discourse. Like, the exact same paper could have instead been titled "Few-Shot Learning with a Large-Scale Language Model" or similar. But instead there seems to be this extremely strong desire to see certain ineffable qualities in neural networks. Like, it's a language model. It does language modeling. Turns out you can use it for few-shot learning and do amazingly well. Beyond that, what does it mean to say it "is" a few-shot learner? On one hand, it's literally the same claim in a strict sense. On the other hand, it implies something much broader and more sweeping, that language modeling / unsupervised learning as a task over long contexts inherently implies meta-learning ability — which is a statement that is very difficult to properly formulate, let alone back up. But that's the argument that I feel is being slipped under the table by these titles. (And indeed it's very close to what they suggest in the text, though with no more than a wave of the hands.) Don't get me wrong: their intuition is reasonable, it's super cool that they got this to work, and the results are very impressive on lots of tasks (though there are clear gaps). But as a VERY publicly watched lab, they have a serious duty (which I think they're neglecting) to frame their results more carefully. In particular, there's a sort of religion that if you train a big enough model on big enough data with self-supervision, it will somehow become AGI and/or learn to solve arbitrary problems. Claims like "Language Models are Few-Shot Learners" are clearly designed to fit into that worldview, even though the research doesn't point at it any more than a more conservative interpretation like "Lots of NLP Tasks are Learned in the Course of Language Modeling and can be Queried by Example." They touch on this limitation in their discussion section but I guess flashy titles are more important. I wish they would use their status to set a better example.
补充展示 GPT-3 生成的新闻文本示例，流畅但可被识破
即使这是 GPT-3 生成的、人类最容易识别出是机器写的文本，我仍然很喜欢它：（附上一篇关于 Joaquin Phoenix 承诺穿礼服、Megyn Kelly 讽刺的假新闻文章全文，内容连贯但细节明显虚构）。
spats1990查看英文原文
Even though this was the GPT-3-generated text that humans most easily identified as machine-written, I still like it a lot: Title: Star’s Tux Promise Draws Megyn Kelly’s Sarcasm Subtitle: Joaquin Phoenix pledged to not change for each awards event Article: A year ago, Joaquin Phoenix made headlines when he appeared on the red carpet at the Golden Globes wearing a tuxedo with a paper bag over his head that read, "I am a shape-shifter. I can't change the world. I can only change myself." It was a promise to not change to fit into the Hollywood mold: "I think that's a really special thing, to not change yourself. I think it's a really special thing to say, `This is what's inside of me, I'm proud of it, and I'm not going to be ashamed because of the way that someone else thinks I should be.'" Now, it's the Oscars, and Phoenix is at it again. But this time, his publicist is saying he'll be wearing a tux no matter what. Megyn Kelly was not impressed, and she let him have it on The Tonight Show. "You know, I feel like, I feel like you could have worn the tux," she says. "But you're saying you're a shape-shifter. I don't know if you can change your tux, but you can change your mind. You can change your mind. You can change your mind." Phoenix says he did, but it didn't stick. "I was like, `Okay, I'm going to wear a tuxedo to this thing.' And then I thought, `I don't want to wear a tuxedo to this thing.'" Kelly goes on to encourage him to change his mind again, but Phoenix says it's too late: "I'm committed to wearing this."
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2020Hacker News 讨论帖431 分 · 201 评
同领域还值得读
DeepSeek-R1：通过强化学习激励 LLM 的推理能力
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
研究如何用强化学习激励大语言模型的推理能力，提出 DeepSeek-R1，并探讨 RL 训练对模型推理表现的影响。
AI 与机器学习2025 · 1,351 分 · 1,056 评 · 引用 30进阶 · 约 80 分钟编辑精选1-bit LLM 时代：面向低成本计算的三值参数
The Era of 1-bit LLMs: ternary parameters for cost-effective computing
提出用三值参数 {-1, 0, 1} 替代 LLM 中的浮点权重，即所谓“1-bit LLM”。
AI 与机器学习2024 · 1,040 分 · 447 评 · 引用 19进阶 · 约 45 分钟编辑精选DeepSeek-V3.2：推进开放大语言模型的前沿
DeepSeek-v3.2: Pushing the frontier of open large language models
这是 DeepSeek-V3.2 的技术报告，延续了 DeepSeek 开源权重模型的路线，目标是让开放模型继续逼近甚至追平前沿闭源模型。
AI 与机器学习2025 · 982 分 · 465 评进阶 · 约 90 分钟编辑精选通过自我对弈与通用强化学习掌握国际象棋和将棋（AlphaZero）
Mastering Chess and Shogi by Self-Play with General Reinforcement Learning
DeepMind 的 AlphaZero 论文：用通用强化学习算法（自我博弈 + MCTS + 神经网络），从零开始仅数小时训练便在国际象棋和将棋上击败了世界顶级引擎 Stock…
AI 与机器学习2017 · 539 分 · 270 评 · 引用 13进阶 · 约 90 分钟编辑精选用强化学习发现更快的矩阵乘法算法
Discovering faster matrix multiplication algorithms with reinforcement learning
论文提出 AlphaTensor，用强化学习搜索矩阵乘法算法。
AI 与机器学习2022 · 518 分 · 111 评专业 · 约 90 分钟编辑精选