网络与分布式2019arxiv.org
Web 缺少一项关键基础设施：开放网络索引
The Web is missing an essential part of infrastructure: an open web index
入门 · 约 30 分钟读完前置知识：搜索引擎工作原理、Common Crawl 基础
导读
这篇论文指出，开放的网络索引（Open Web Index）是 Web 基础设施中缺失的一环：目前高质量的全网索引被 Google、Microsoft 等少数商业搜索引擎垄断，研究者和小公司难以获得。作者主张建立一个开放、可公开访问的网络索引作为公共基础设施，让搜索创新不再受制于大公司。
HN 关注点：HN 读者长期关注搜索引擎垄断与开放基础设施话题，讨论焦点集中在 Common Crawl 等现有项目是否已经实现了论文的主张，以及 Internet Archive 等机构能否承担这一角色。
HN 评论精选
这几条评论里：质疑 1 补充 3
补充指出 Common Crawl 已在做类似的事，并引 DuckDuckGo 创始人关于索引应独立于搜索引擎的观点
这不就是 Common Crawl 吗？它是一个非营利组织，致力于免费向研究者、公司和个人提供一份互联网的副本，用于研究分析。人们用它改进机器翻译、预测趋势、追踪疾病传播等。他们的目标是把数据民主化，让不只是大公司才能做高质量研究。另外 DuckDuckGo 创始人多年前就表达过索引应与搜索引擎分离的观点：把“复制互联网”当作商品，可以从多个渠道获取。不过我记得 DuckDuckGo 并不使用 Common Crawl。
weinzierl查看英文原文
Isn't this what Common Crawl[1] is. From their FAQ: > What is Common Crawl? > Common Crawl is a 501(c)(3) non-profit organization dedicated to providing a copy of the internet to internet researchers, companies and individuals at no cost for the purpose of research and analysis. > What can you do with a copy of the web? > The possibilities are endless, but people have used the data to improve language translation software, predict trends, track the disease propagation and much more. > Can’t Google or Microsoft just do that? >Our goal is to democratize the data so everyone, not just big companies, can do high quality research and analysis. Also DuckDuckGo founder Gabriel Weinberg expressed the sentiment that the index should be separate from the search engine many years ago: > Our approach was to treat the “copy the Internet” part as the commodity. You could get it from multiple places. When I started, Google, Yahoo, Yandex and Microsoft were all building indexes. We focused on doing things the other guys couldn’t do. [2] From what I remember reading once DuckDuckGo doesn't use Common Crawl though. [1] https://commoncrawl.org/ [2] https://www.japantimes.co.jp/news/2013/07/28/business/duckduckgo-chief-spills-on-search-engine-wars/
补充梳理 Common Crawl 与 Mixnode 两个现有项目的差异与局限
目前有两个实体在尝试做这件事：Common Crawl（非营利）按月进行广泛的常规抓取，以 WARC 文件存储，并提供一个可用于查询的独立索引（但不是全文索引），主要用于学术界；Mixnode（营利）定期抓取网页并允许用户用 SQL 查询数据，目前处于内测。还有一些搜索引擎 API，但我认为利益冲突使其难以提供经济高效的大规模访问和定价。
netborn查看英文原文
There are two entities trying to pull this off: Common Crawl (non-profit): Stores regular, broad, monthly crawls as WARC files. Provides a separate index that can be used to look data up (no a fulltext index though). Used mostly in academia. Mixnode (for-profit): Regularly crawls the web and lets users write SQL queries against the data. Not sure who the primary users are since it's in private beta. There are some search engine APIs, but I don't think the conflict of interest would allow for cost-effective large-scale access and pricing...
补充建议以 Internet Archive 为基础构建开放索引
Internet Archive（web.archive.org）能否成为作者所提议的开放网络索引的基础？这条路上肯定有很多障碍，但相比任何新项目它至少有两点巨大优势：它已经拥有庞大的索引和采集流水线，而且是一个值得信赖的机构。
mlinksva查看英文原文
Could the Internet Archive, specifically https://web.archive.org/ be the basis of an Open Web Index as proposed by the author? I'm sure there are tons of obstacles to that path, but it also would be far ahead of any new initiative in at least two ways: it already has a huge index and ingestion pipeline, and it is a trusted organization.
质疑主张用 P2P 和多个领域专用索引替代单一元索引
论文似乎是在推荐 Open Web Index（它有自己的网站）。我更喜欢一个改良版本：它应该是 P2P 技术，不试图建立一个单一的元索引，而是建立许多领域专用的索引，配合一个或多个工具/数据库根据查询和上下文选择要搜索的索引。另外这与从以服务器为中心的互联网转向以内容为中心的互联网的想法有重叠。
ilaksh查看英文原文
It seems like the idea is recommending the Open Web Index (has its own website). I like a modified version of this. I think that it should be a p2p technology and not try to create one meta-index but rather be many domain-specific ones, with one or more tools or DBs to select which indices to search given a query/context. Are there any decentralized alternatives to Google out there already? I think that also this overlaps with the idea of moving from a server-centric internet to a content-centric internet.
相关资源
从这篇论文所有 HN 讨论帖的全部评论里整理出来的链接。
历次讨论
- 2019Hacker News 讨论帖548 分 · 129 评
同领域还值得读
Paxos vs. Raft：分布式共识是否已达成共识？
Paxos vs. Raft: Have we reached consensus on distributed consensus?
重新审视了 Paxos 与 Raft 两种分布式共识算法之间的差异，试图在一个统一的框架下比较它们，并论证二者的相似程度远高于通常的认知，许多所谓的差别只是表述层面的。
网络与分布式2020 · 343 分 · 65 评 · 引用 10进阶 · 约 28 分钟编辑精选Stellar 协议：互联网级共识的联邦模型
Stellar Protocol: A Federated Model for Internet-Level Consensus (2016)
提出 Stellar Consensus Protocol（SCP），一种面向互联网级共识的联邦拜占庭协议模型。
网络与分布式2018 · 301 分 · 61 评专业 · 约 90 分钟编辑精选拜占庭将军问题（1982）
The Byzantine Generals Problem (1982)
这篇 1982 年论文提出并形式化了拜占庭将军问题：在分布式系统中，部分节点可能叛变或故障、通信不可完全信任时，忠诚节点如何达成一致行动。
网络与分布式2018 · 290 分 · 77 评进阶 · 约 75 分钟编辑精选Google 变差了吗？对搜索引擎中 SEO 垃圾的纵向调查
Is Google Getting Worse? A Longitudinal Investigation of SEO Spam in Search
对搜索引擎结果中的 SEO 垃圾内容做了纵向追踪：作者团队在一段时间内持续采集 Google、Bing、DuckDuckGo 等引擎针对商品评测类查询的结果，用分类器识别联盟营销…
网络与分布式2024 · 276 分 · 274 评入门 · 约 45 分钟编辑精选使 CRDT 能够容忍拜占庭故障
Making CRDTs Byzantine Fault Tolerant
Martin Kleppmann 等人的论文研究如何让 CRDT（无冲突复制数据类型）抵御拜占庭故障节点，即部分参与者恶意篡改数据时仍能保证一致性。
网络与分布式2022 · 266 分 · 40 评进阶 · 约 90 分钟编辑精选