Does EtherPad send all Changesets for huge files?

Question

Suppose there is a huge file where the number of changes is gigantic so that the collection of changesets is gigantic. Does EtherPad send all Changesets? Or does it periodically take a snapshot of the file so that it can avoid sending the really really old changesets? And how is this all implemented in the backend database?

Answer 1

RE sending changesets see: https://github.com/ether/etherpad-lite/tree/master/doc/easysync

There is no file to take snapshots of so I assume you mean the pad contents? If so, no, Etherpad Lite doesn't save meta text yet but this is something Etherpad will probably need to implement so that the timeslider can handle pads with large content (see the relevant issues that discuss this). The old Etherpad did have this functionality but it came at a cost (Horrible database schema).

The Etherpad database is abstracted through something called UeberDB https://github.com/Pita/ueberDB. The big win UeberDB gives Etherpad is a caching layer in front of writes and portability. UeberDB empowers Etherpad to hold pad contents in memory and doesn't require the overheads of constant database writes, writes are queued as to reduce server load. UeberDB also allows site admins to use their favourite database but at a cost of having to store KVs.

RE points 2 and 3, because Ueber will only store key values you end up struggling doing tasks such as Full text indexing which is a key feature of MySQL despite using MySQL as a backend database. This has however been recently overcome and is in a branch partially maintained by Mozilla and various other Etherpad project contributors. Also MySQL provides it's own caching functionality however this isn't useful for Etherpad as the data is in a constant state of flux.

We should be clear I have been talking about Etherpad Lite as the "old Etherpad" is no longer maintained so should be avoided. All Etherpad documentation should be referenced to from the authoritative source of http://etherpad.org