itty-bitty icon indicating copy to clipboard operation
itty-bitty copied to clipboard

Consider using different compression (brotli)

Open phiresky opened this issue 7 years ago • 6 comments

Just wanted to let you know that brotli significantly improves compression for this usecase, probably mostly because of brotli's integrated dictionary.

For the landing page:

  • Uncompressed: 3623 chars
  • Your LZMA version: 2292 chars
  • Brotli + Base64: 1768 chars

Declaration of independence:

  • Uncompressed: 8914 chars
  • Your LZMA version: 5168 chars
  • Brotli + base64: 4184

A JS implementation of brotli is here.

phiresky avatar Jul 06 '18 13:07 phiresky

Interesting. I'll look into it :)

alcor avatar Jul 06 '18 14:07 alcor

The only real trade-off is the size of that dictionary - it is nearly 1MB, which becomes expensive to serve over many loads. lzma had the benefit of being small, ~10k. I'll try it as an alternate mode, though.

alcor avatar Jul 06 '18 14:07 alcor

Looks to me like the full official decoder (js version) is only 150 kB (70kB compressed)

check out this complete example: https://gist.github.com/phiresky/ff29eeef882aaf002d7b747ec04f6113

phiresky avatar Jul 06 '18 14:07 phiresky

The decoder is easy, but looks like it's harder to find a working, compact JS encoder for brotli without external dependencies.

phiresky avatar Jul 06 '18 15:07 phiresky

Yeah, I'm trying to get one, but running into this bug: https://github.com/devongovett/brotli.js/issues/20

alcor avatar Jul 06 '18 15:07 alcor

Just for fun, I compiled the Rust Brotli implementation to WebAssembly. It works pretty well, the encoder is 970kB (470kB compressed).

See here: https://phiresky.github.io/rust-brotli-wasm/brotli-wasm-bindgen/dist/example.html (code)

No idea if all browsers support webassembly yet though, and I'm too lazy to transpile it down to asm.js.

phiresky avatar Jul 06 '18 21:07 phiresky