hackademix
Syncing large whitelists fails
Since Firefox Sync only permits 16384 byte objects to be saved, NoScript whitelists can quickly become unsyncable.
Quoting myself from https://forums.informaction.com/viewtopic.php?f=7&t=6056&p=96391#p96391
Workaround idea:
- serialize data-to-be-synced to json
- compress it ( see e.g. http://forums.mozillazine.org/viewtopic.php?p=14111285#p14111285 )
- base64 encode it
- split into 16384 B chunks (minus a few bytes for a sync timestamp + part number headers)
- when reading, combine chunks based on headers
- decompress
- deserialize.
That should get at least a few hundreds of kilobytes of worth of uncompressed config synced.
If there is interest in something like this, but no time to implement, I would be interested in contributing.
Hi, thank you for the idea. Actually the proposed compression method won't fly, because it uses "legacy" APIs which are not available to WebExtensions. However the general concept still applies, and your code contribution would be more than welcome :)
I haven't looked at the code very much but do you think it would make sense to do this entirely in noscript/src/common/Storage.js ? maybe along the following lines? Of course checking which is a currently good way of doing things.
set: If data size is above a treshold
- Serialize
- Compress
- Split into part-objects
- Save parts
- Save a metadata object containing list of chunks that are required for reassembly
get: If data is a metadata object:
- Read which objects are required for reassembly
- get them
- combine parts
- decompress
- deserialize
- return reassembled object ?
If data size is above a treshold
Is there a hard limit on the total amount of data your can sync for all of your installed extensions?
If so
The threshold, if there is one, should be whichever is the smallest: a compressed version and a non-compressed version.
That is, if you're going to compress you might as well compress it if compressing will make it smaller overall for the sake of other extensions to leverage the free sync space.
My 2 cents: Why not using p2p between devices for whitelist sync and only necessary p2p data sync with FF Sync? I know this is out of pure firefox API sync process and won't store whitelist data on FF sync server (so no backup) but at list, we can sync whitelist...
