keyv
keyv copied to clipboard
jaredwray
Alternative compression methods
Is your feature request related to a problem? Please describe.
I'm using Keyv to wrap connection with Redis and I would like to compress the content cached in Redis. This caching process is time-sensitive, so it should favour speed over compression rate. Thus why I'm atm using separate wrapper using snappy as the compression algorithm. I did performance tests and it turned out that Brotli in BROTLI_MIN_QUALITY mode is almost 2x slower than snappy!
So, the real problem for me is that Brotli is the built-in compression mechanism that cannot be easily changed for anything else.
Describe the solution you'd like
I would like to have some options if it goes to choosing the compression method that can be used with Keyv.
Describe alternatives you've considered
Atm I'm using custom kdybicz/apollo-server-snappy-cache-wrapper wrapped for Keyv.
Additional context
Scotty, We Need More Power!
@kdybicz - Thanks for sending this over and we are also looking to make the compression engine more modular for Keyv moving forward. This would make it so that we can be more browser friendly (https://github.com/jaredwray/keyv/issues/252). Some thoughts on this is to create a generic interface for these modules that would interact more like this:
const Keyv = require('Keyv');
const KeyvBrotli = require ('KeyvBrotli')
const keyvBrotli = new KeyvBrotli();
const keyv = new Keyv({ compression: keyvBrotli });
This would make it more flexible to add in other libraries and could be supported via a standard interface. Would like your thoughts on this.
This sounds very interesting and I guess the compression can also be optional then which would be 1 less module to load in lean environments like a Cloudflare worker or browser
Yep. much more scalable solution. We hope to have those changes out in the next 30 days or so then will get back to making it browser compatible.
@jaredwray I really like that idea. Such approach would also make the configuration of the compression more clear, as it would be delegated to a dedicated module, one with a clear (I hope) interface :)
@alfaproject and @kdybicz - we will begin this work in August and plan to release the change in September 2022. Will post a description of the plan in this issue.
@alfaproject and @kdybicz we have merged in the changes and will be planning a release this week. Also, will add in support for snappy as a compression provider and most likely gzip.
@jaredwray snappy integration is something you can probably ignore for now, as it might be tricky to manage due to native libraries use by it and is not really needed by me anymore. It was way faster, but it turned out I need bigger compression ration that it was providing. So I've created my own compression wrapper for brotli which for some reason happens to be way faster than one that was built-in - no idea why though, maybe I messed up the configuration. So atm I'm using brotli and would be happy to switch to keyv dedicated solution, rather using custom one. At the same time adding gzip might make sense.
Some time ago I've found this compression methods benchmark, not sure if you will find it helpful https://quixdb.github.io/squash-benchmark/
@kdybicz - great will focus on the gzip version next using most likely as we want to support at least two. Will get that going and most likely release all of this update this upcoming weekend.
@jaredwray looks like you might need to secure the project bit better 👆🏼 (edit: the rouge comment that was sent via email has vanished as the account of the :trollface: was deleted)
@jaredwray looks like you might need to secure the project bit better 👆🏼 (edit: the rouge comment that was sent via email has vanished as the account of the :trollface: was deleted)
Crazy that this happened. Wonder what it was. There have been no pull requests done or code submitted so I think the project is ok.
@jaredwray looks like you might need to secure the project bit better 👆🏼 (edit: the rouge comment that was sent via email has vanished as the account of the :trollface: was deleted)
Crazy that this happened. Wonder what it was. There have been no pull requests done or code submitted so I think the project is ok.
He mailed-in that comment and I though there is a setting in project configuration to enable/disable that functionality, but I cannot find anything in my projects o.O
We have now brought in support for gzip with this PR https://github.com/jaredwray/keyv/pull/457
Plan will be to do the breaking update end of week
Closing this issue as we have now published the compression changes. https://github.com/jaredwray/keyv/releases/tag/2022-09-02