安全与密码学2022arxiv.org编辑精选
在机器学习模型中植入不可检测的后门
Planting Undetectable Backdoors in Machine Learning Models
专业 · 约 80 分钟读完前置知识：机器学习基础、密码学假设、计算复杂性理论
导读
这篇论文研究恶意训练者能否在机器学习分类器中植入“不可检测”的后门：模型表面上表现正常，但持有密钥的攻击者只需对输入做轻微扰动，就能把任意样本改成自己想要的分类。作者给出两种后门构造框架，并声称在没有后门密钥的情况下，任何计算能力有限的观察者都无法检测到该机制；其不可检测性依赖密码学假设。
HN 关注点：外包或第三方训练的模型是否还能被信任，以及仅靠输入输出测试或梯度后处理能否发现后门。
后来怎么样
将密码学中的不可检测性概念引入机器学习后门研究，启发了后续关于后门检测极限与可验证训练的理论工作。
HN 评论精选
这几条评论里：支持 1 质疑 1 补充 2
支持用直观例子说明后门可藏在特定触发条件中，且外部模型难以完全信任。
总体看这挺直观：如果我提供一个能识别汽车的模型，而我知道你把它用在测速摄像头上，我就可以把模型训练成一切如你所愿，唯独车窗上贴有“GTEHQ”贴纸的汽车不被识别。这样我就有了一个你不知道、却能影响模型的后门。可以想象，要从模型逆向工程出这种训练痕迹会非常非常困难，用测试也很难发现——你怎么会想到测试这个特定情况？同样的事也能对许多其他模型做。除非你能自己训练模型，否则很难 100% 信任别人训练的模型。
sponaugle查看英文原文
Overall this seems somewhat intuitive - If I offer to give you a ML model that can identify cars, and I know you are using it in a speed camera, I might train the model to recognize everything as you expect with the exception that if the car has a sticker on the window that says "GTEHQ" it is not recognized. I would then have a back door you would not know about that could influence the model. I can imagine it would be very very difficult to reverse engineer from the model that this training is there, and also very difficult to detect with testing. How would you know to test this particular case? The same could be done for many other models. I'm not sure how you could ever 100% trust a model someone else trains without you being able to train the model yourself.
补充解读具体构造，并指出实际模型结构检查可能发现并行后门，但核心结论是仅靠查询不可判定。
他们给出的一个构造可概括为：在已有神经网络旁并联一个只需几层的小网络，根据输入中隐藏的某些信息（例如几个输入值的符号）做密码学签名验证；再接一个最终层，根据验证是否成功，要么输出原模型结果（签名无效），要么输出攻击者选定的结果（签名有效）。如果巧妙利用消失梯度，甚至（或可以做到）对受害者后续再训练保持鲁棒。把这种方法用到实际 ML 模型当然是留给读者练习。研究确实证明这根本上是可能的（数学上也很简单），但我觉得多数实际应用中 ML 模型结构相对透明，因此这样构造的“并行”验证网络较容易被发现：数据流图会很暴露，只是受害者得真的去检查。那这又变成混淆游戏——能否把签名检查和最终选择步骤不显眼地藏进主网络？我相信只要足够决心总能找到办法。但我认为论文最突出的点是：1）仅凭输入-输出查询无法判断模型是否被植入后门（除非已经知道后门）；2）这意味着……
MauranKilom查看英文原文
To summarize one of the constructions they provide: If you have an existing neural network, you can add a parallel network with just a few layer that performs cryptographic signature verification based on something hidden in the input (e.g. signs of a few input values). Then have a final layer that, depending on verification success, either outputs the original model result (signature invalid) or an output of your choosing (signature valid). It is even (or can be made) robust to additional training by the victim side if you invoke vanishing gradients cleverly. Applying this to a practical ML model is of course left as an exercise for the reader. While the research certainly proves that it's fundamentally possible (and mathematically trivial) to do such a thing, I feel that the structures of ML models are relatively transparent in most practical applications, making it comparatively easy to detect "parallel" verification networks thus constructed. The dataflow graph will be pretty revealing, but the victim would have to actually inspect it in the first place. Of course, that in turn makes it a game of obfuscation - can you inconspicuously hide the signature check and final muxing step among the main network? I have no doubts that you can find a way if you're so determined. But I think the most salient points of the paper are that 1. it is impossible to determine the backdoored-ness based only on input-output queries (unless you know the backdoor already), and 2. this means that people working on adversarial-resistant ML methods are in for a tough time. There's more to be found in the paper, this is just my short summary after reading the most interesting bits.
质疑强调 MLaaS 使后门风险商业化，并质疑其最短向量问题密码学假设。
（免责声明：我只略读了文章，已列入待读。）我第一次接触对抗样本时以为这是小众担忧。但正如这篇论文所述，机器学习即服务公司（Amazon、OpenAI、Microsoft 等）的增长已让它成为真实担忧。略读中我想强调他们一个有趣观点：“基于梯度的后处理”在缓解被污染模型方面可能作用有限。这些点把担忧从学术圈带到商业领域。最后，我很高兴他们承认受密码学社区影响，严谨量化“困难”和“不可区分”等概念。值得注意的是，他们似乎把不可检测后门建立在“最短向量问题不在 BQP”这一假设上；我最近从 NIST 后量子风波中学到，这一点争议很大。我很可能误述了论文，但期待细读。
kvathupo查看英文原文
(Disclaimer: I skimmed the article, and have it on my to-be-read) When I first encountered the notion of adversarial examples, I thought it was a niche concern. As this paper outlined, however, the growth of "machine-learning-as-a-service" companies (Amazon, Open AI, Microsoft, etc.) has actually rendered this a legitimate concern. From my skimming, I wanted to highlight their interesting point that "gradient-based post-processing may be limited" in mitigating a compromised model. These points really bring these concerns from an academic to business realm. Lastly, I'm delighted that they acknowledge their influences from the cryptographic community with respect to rigorously quantifying notions of "hardness" and "indistinguishable." Of note, they seem to base their undetectable backdoors on the assumption that the shortest vector problem is not in BQP. As I recently learned looking at the NIST post-quantum debacle, this has been a point of great contention. I've in all likelihood mischaracterized the paper, but I look forward to reading it!
补充引用摘要：后门模型表面正常，持密钥者可轻微扰动改判，且对计算有限观察者不可检测。
“……我们展示恶意学习方如何在分类器中植入不可检测后门。表面上看，这种被植入后门的分类器行为正常，但实际上学习方保留了一种机制，只需轻微扰动就能改变任意输入的分类。重要的是，没有相应的‘后门密钥’，该机制就是隐藏的，任何计算能力有限的观察者都无法检测。我们展示两个植入不可检测后门的框架，其保证不可比较……” PDF: https://arxiv.org/pdf/2204.06974.pdf
belter查看英文原文
"...We show how a malicious learner can plant an undetectable backdoor into a classifier. On the surface, such a backdoored classifier behaves normally, but in reality, the learner maintains a mechanism for changing the classification of any input, with only a slight perturbation. Importantly, without the appropriate “backdoor key,” the mechanism is hidden and cannot be detected by any computationally-bounded observer. We demonstrate two frameworks for planting undetectable backdoors, with incomparable guarantees..." PDF: https://arxiv.org/pdf/2204.06974.pdf
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2022Hacker News 讨论帖275 分 · 59 评
同领域还值得读
如何入侵一台已关机的电脑：在 Intel ME 中运行未签名代码
How to hack a turned-off computer, or running unsigned code in Intel ME
这篇 Black Hat 2017 白皮书讨论 Intel Management Engine（ME）的安全问题：如何在 ME 中运行未签名代码，以及攻破处于关机状态的电脑的相关…
安全与密码学2017 · 543 分 · 107 评专业 · 约 40 分钟编辑精选比特币：一种点对点的电子现金系统（2008）
Bitcoin: A Peer-to-Peer Electronic Cash System (2008)
中本聪 2008 年发表的比特币白皮书，提出了无需可信第三方的点对点电子现金系统。
安全与密码学2017 · 536 分 · 394 评入门 · 约 45 分钟编辑精选GPRS 加密算法 GEA-1 的密码分析表明存在故意弱点
Cryptanalysis of GPRS Encryption Algorithms GEA-1 suggest intentional weakness
首次公开了对 GPRS 加密算法 GEA-1 和 GEA-2 的密码分析攻击。
安全与密码学2021 · 487 分 · 107 评专业 · 约 90 分钟编辑精选Signal 消息协议的形式化安全分析
A Formal Security Analysis of the Signal Messaging Protocol
用形式化方法分析 Signal 消息协议（TextSecure）的安全性：作者在多阶段密钥交换模型下，借助自动化证明工具对协议的密钥协商与 Double Ratchet 密钥更新…
安全与密码学2016 · 353 分 · 230 评专业 · 约 120 分钟编辑精选A2：模拟恶意硬件
A2: Analog Malicious Hardware
论文研究如何在芯片设计阶段植入模拟电路形式的恶意硬件后门，使其难以被常规检测发现。
安全与密码学2016 · 315 分 · 94 评专业 · 约 75 分钟编辑精选