Use lz4.frame

[bmoscon]

Its not that hard to exceed lz4 memory constraints or or cause it to overflow. The solution is to use lz4.frame which lets you compress a block of data in chunks.

[fl4p-old]

Isn't block size capped to 16MB anyway due to mongodb constraint?

[bmoscon]

true, but data is serialized, compressed and then broken up into 16MB chunks for storage in mongo, so the size constraint is currently on Lz4's block size, which is why we should move to frame.

[bartmostert1]

Every time i want to use arctic i get the following errror

No module named 'lz4._version'

already removed arctic and re install it throw pip but still it's not working

[bmoscon]

@bartmostert1 that seems unrelated to this feature request. Please open a new ticket and specify the operating system and give the ouput of pip freeze