Can fullproof update its full-text index without a complete re-index?

Question

I'm evaluating the fullproof javascript full text search engine, but I can't figure out if it can handle index updates. From my reading of the api I don't see the mention of updating or deleting something from the index.

lunr.js for example can make updates to the index without a complete reindex.

I hope it can make index updates, because on my iPad, it took 10 minutes to create the index, which is prohibitive if it has to rebuild the index every time. Compared to 40 seconds to create the lunrjs index. However the problem with lunr is that the complete index needs to be in memory, whereas in fullproof it puts the index in a local WebSQL db.

Answer 1

It looks like fullproof can't update/delete items from it's index given the lack of comments on this github issue.

Answer 2

I can't speak for fullproof, but there is nothing stopping you from snapshotting the lunr index into client side storage yourself.

Given the following index:

var idx = lunr(function () {
  this.ref('id')
  this.field('title')
})

You can dump it as JSON like this:

var dump = JSON.stringify(idx)

Now you can store that dump anywhere you like, I'm not overly familiar with WebSQL but storing it in localStorage might look like this:

localStorage.setItem('lunr_dump', dump)

To reload the index you do the following:

var idx = lunr.Index.load(dump)

Loading a dumped index back into lunr should be several orders of magnitude quicker than rebuilding it each time.

As you mention, lunr doesn't need to re-build the whole index each time, so lets say you update a document, and would like to keep the dumped version up to date:

idx.on('update', function (doc, index) {
  localStorage.setItem('lunr_dump', JSON.stringify(index))
})

idx.update(someDocument)

This will dump the entire index again, including any changes made when updating. It is worth benchmarking the dump, as there might be some overhead, especially with larger indexes.

If your data is relatively static you can actually pre-build an index on the server and send this to the client, saving the client the work of building the index on each page load, take a look at https://github.com/olivernn/lunr-index-builder

You can take this slightly further though, since the dumped index is just JSON, there is nothing stopping you indexing documents individually and then later merging the dumped indexes together, there is an issue here with some discussion and a link to a blog post about implementing it if your interested.