Open Web Indexing

Here are two approaches to making an open index of the web. One is the Common Crawl Foundation. The other is YaCy.

Common Crawl Foundation
http://www.commoncrawl.org/data/accessing-the-data/
http://www.i-programmer.info/news/136-open-source/3320-common-crawl.html
http://www.commoncrawl.org/common-crawl-enters-a-new-phase/
http://search.slashdot.org/story/11/11/15/0057200/common-crawl-foundation-providing-data-for-search-researchers

YaCy:
YaCy users make nodes that can provide indexing services and search engine query services. YaCy indexing is decentralised and portions of the index are redundantly distributed. Individual nodes can provide both private and public indexing and serve search requests. YaCy can also be used to make a private index of an intranet.
http://yacy.net/en/
http://search.yacy.net/ (demo search portal)
http://tech.slashdot.org/story/11/11/28/224251/free-software-activists-take-on-google-search

http://yacy.net/en/Applications.html