Decentralized Search in 2020

Parking ideas for a decentralized search engine from first principles in 2020

Goal - to build a new age search engine (fun project) that is semantic and decentralized.

Some initial thoughts

  • Electron UI
  • Modeling
    • Semantic modeling (schema.org)
    • Ability to add / edit models
    • Ability to add / edit objects
  • Data Extraction
    • Source Plug-in architecture (example Wikipedia plugin, StackOverflow plugin, Reddit plugin, Amazon plugin etc.).
    • The source plugins will be extract information and map it semantically in a local SQLite database.
  • View
    • Multiple object view. with plugins for rendering objects as list, graphs, images
    • Single object view. with templates for each object type and also showing modeled objects with internet links for further search
  • Sharing
    • Ability to publish / share models, objects, extractors, views in-app
  • Network
    • Offline first
    • Personal node first
    • Personal node can be setup as a server
    • Ability to create a peer network (node to node)
    • Ability to search a peer via a plugin

Reading

Pears: https://pearsearch.org/

Blog on personal search: https://beepb00p.xyz/pkm-search.html

4 Likes

Interesting idea. Seems more like a knowledge store/graph than a search engine. SQLite / or an RDBMS probably wouldn’t be the best way to model semantic relationships. A graph DB would be better suited.

Speaking of knowledge graphs, check out https://cayley.io and https://perkeep.org/ (previously Camlistore. Not a knowledge graph but a personal “life store”).

An interesting project is https://connectordb.io/ - intended to be a personal Google Analytics equivalent (known as “Quantified Self”). It has become dormant but they did make a number of data collection software for various platforms (browsers, OS).

A social search (powered by fossasia) https://loklak.org

If we talking about a search engine, then we possibly don’t need a Electron UI for this.
Moreover, we might be able to leverage IPFS for distributed storage needs as well.
(I have yet to figure out on a model on how this will actually work, but for e.g

Every node gets preference if they help us index the site

Just like if we seed files (in a torrent based Architecture) we get more download speeds.
That’s where the Search Engine Algorithm comes, I mean ok we are taking it as a “fun” project but I think engineering this is important as well.

I also realize there are 2 steps to “freeing” a service

  1. Make it community owned
  2. Make it distributed

While we jump to #2, we should not forget #1. This is where we hope FOSSUnited can be a platform to manage community owned versions of popular services like search, social, market etc.

Also #1 is much easier to do given the current internet infra. Also much easier to sustain and moderate content on community owned platforms.

1 Like