numcodecs.js icon indicating copy to clipboard operation
numcodecs.js copied to clipboard

Compare to other implementations

Open domoritz opened this issue 3 years ago • 2 comments

We are looking at lz4 implementations (and possibly other compressions) for Apache Arrow in https://issues.apache.org/jira/browse/ARROW-8674 and it would be very interesting for us to understand how this library compares in terms of bundle size and speed with https://github.com/gorhill/lz4-wasm, https://github.com/Benzinga/lz4js, and https://github.com/pierrec/node-lz4. Have you looked into these or have any thought on how they might compare?

domoritz avatar Apr 19 '22 02:04 domoritz

This is a great question, and the short answer is I'm not sure. I created this library for https://github.com/gzuidhof/zarr.js/ and wanted pure ESM implementations of different codecs. Originally I'd seen pierrec/node-lz4, which offered way more features than I required and additionally (I believe) relies on a Buffer polyfill in the browser.

I hadn't seen the other two implementations you mentioned. My guess is that https://github.com/Benzinga/lz4js is likely the smallest in terms of bundle size along with https://github.com/gorhill/lz4-wasm. Minified, the lz4 codec in this repo is very big at ~40kb. In terms of performance, I haven't done any profiling or comparison.

manzt avatar Apr 19 '22 16:04 manzt

Thank you. That's very useful.

domoritz avatar Apr 19 '22 17:04 domoritz