Extension to arbitrary number of channels

[alejoe91]

Hi WavPack developers,

First of all thank you for the project!

I'm writing with an unusual use case. I'm an electrophysiology data engineer and work with signals recorded from inside the brain with "so-called" high-density probes. Just to set some context, these probes can have over 1000 recording sites with ~15um space between each other, for this is a "Neuropixels" probe :

It might seem funny, but neural data recorded by these devices are kind of similar to audio signals in the sense that:

• they share more or less the same frequency content
• they are sampled at similar rates (in our case 30kHz)
• there is a high redundancy between channels

We are looking at ways to compress these data and audio codecs seem to to a great job (we get CR>3 out of the box). Most of our dataset record from 384 channels simultaneously and this number is expected to grow over the next years. So my first question is:

1. would it be possible / a lot of work to extend the codec to support any number of channels (basically any 2D matrix)?

A second and related problem is how to make wavpack a "general" compressor for any 2D data. We are currently using zarr, which provides a very user friendly python interface and wraps several compressors. Implementing a new Zarr-compatible compressor seems relatively easy (but would probably require some Python bindings).

2. Would you guys be interested in such an option? I'm happy to discuss more about it in case you are!

(Full disclosure: I started playing around with FLAC and they pointed me to WavPack)

thank you in advance!

Alessio

[dbry]

Hi Alessio,

Interesting application! Actually a good friend of mine worked at Neuralink for a couple years, and we discussed this same thing several times!

The WavPack compression library (libwavpack) supports an almost unlimited number of channels (4096 or 8192) already, so as far as I know there's nothing else to do there.

The command-line programs are limited to 256 channels, but that is only in there to have something reasonable and testable. It would be trivial to remove or increase that limitation.

As for the "general" compressor, it sounds interesting if other people would find it useful. I am no longer working actively on adding features or new functionality to WavPack, but I can certainly answer questions and provide help as needed.

Another thing that you might possible consider is using the lossy or hybrid modes of WavPack to save even more space. Unlike a fixed quantization level, the lossy mode dynamically scales the quantization with the signal level, so if you are utilizing 16-bit (or whatever) data because of a dynamic range requirement, you may be wasting resolution at higher signal levels and the WavPack lossy mode might be useful (it is not based on psychoacoustics, so it doesn't attempt to discard anything it considers "inaudible").

Thanks the the query, and let me know if this didn't completely answer your questions, or you have something else.

Kind regards, David

[alejoe91]

Hi David,

Glad to hear that we don't need ay modification to extend it beyond 256 channel! :) Could you point mo to the code that needs to be changed in otder to do so? So I can rebuild and test it locally.

For the "general" puprose compressor, I can try to bind the C libraries to Python and wrap them in numcodecs. Is there a specific C function to compress and decompress arrays to flat binary data?

I'll definitely try the hybrid mode as well! Indeed, lossy compression is also something we're looking at and we're exploring different modalities! Thanks for the tip!

Alessio

[alejoe91]

@dbry I wanted to try to use the hybrid mode via the CLI, but I cannot get it to work:

>>> wavpack mywav.wav -bn 16 -o test_hybrid.wv

Gives:

hybrid spec must be 2.0 to 9600!

Isn't the hybrid spec the one specified by the -bn option?

Thanks for the help! Alessio

[dbry]

Glad to hear that we don't need ay modification to extend it beyond 256 channel! :) Could you point mo to the code that needs to be changed in otder to do so? So I can rebuild and test it locally.

It depends on how you're using the compressor. If you are using the --raw-pcm mode then the check is here:

https://github.com/dbry/WavPack/blob/4fc3da677609d297eb38e0d92bf00d16f36d9949/cli/wavpack.c#L456

If you are doing this from WAV files, then the check is here:

https://github.com/dbry/WavPack/blob/4fc3da677609d297eb38e0d92bf00d16f36d9949/cli/riff.c#L189

The other codecs are similar and should be easy to find.

For the "general" puprose compressor, I can try to bind the C libraries to Python and wrap them in numcodecs. Is there a specific C function to compress and decompress arrays to flat binary data?

There isn't really something so straightforward, unfortunately, to just convert from one buffer to another. The "compress" functions use callbacks for providing the compressed data to the client. For decompression, there is a function called WavpackOpenRawDecoder() that might be useful. I would suggest looking in the library documentation and we can discuss further:

https://github.com/dbry/WavPack/blob/master/doc/WavPack5LibraryDoc.pdf

[dbry]

@dbry I wanted to try to use the hybrid mode via the CLI, but I cannot get it to work:

>>> wavpack mywav.wav -bn 16 -o test_hybrid.wv

Gives:

hybrid spec must be 2.0 to 9600!

Isn't the hybrid spec the one specified by the -bn option?

Thanks for the help! Alessio

Sorry that's not clearer, but the 'n' is the decimal bitrate, so your example would be just -b16, and it can also be non-integer, like -b2.5.

You mentioned that you're getting already better than 3:1 compression, which I assume is with 16-bit data, which means you're already compressing down to around 5 bits per sample losslessly. I would suggest starting around 3 or 4 bits per sample for the lossy mode. The minimum that the lossy mode will do is around 2.25 bits per sample.

Of course, keep in mind that the value is an "average" target bitrate. Individual samples will use a variable number of bits to achieve the target bitrate while still preserving detail in transients (which I suspect you're interested in).

[alejoe91]

@dbry thank you very much! The hybrid mode is running smoothly now and now I can process all channels at once!

Should I make a PR where one can specify a constant with the MAX_NUM_CHANNELS?

Thanks for sharing the library doc! I'll read through it and we can discuss how to proceed. For now, I was able to have something working by using tmp wav and wv files. Seems to work ok and also be relatively fast, but I'm sure we can find a better solution for it :)

[dbry]

@alejoe91 I'll put in the change to create a constant for WAVPACK_MAX_CLI_CHANS; that seems like a good idea.

I was also looking over the code and see that maybe there's an issue with over 4096 channels, so that might be the actual upper limit. Will investigate.

[alejoe91]

Thanks! :)

[dbry]

I have verified that the maximum channel count is actually 4096 and I have updated the code to reflect this and prevent any attempts to go beyond this limit.

Please feel free to let me know if you run into anything else!

[alejoe91]

Hi David,

We have been testing WavPack (both lossless and hybrid mode) and the compression results are very promising!

As I mentioned, we wanted to include WavPack as a codec in the Zarr library and we have a "working" implementation that you can find here. For testing purposes, the compression implementation currently goes from a numpy buffer --> wav file (using scipy) --> wv with CLI (and the other way around for decompression). While compression speed is quite fast, decompression is much slower and I think that it could be because of the "convoluted" implementation.

Would you have time to have a chat and discuss ways to possibly make things smoother?

Thanks again! Alessio

[janstary]

Alessio, is it possible to get a sample of these data, i.e. a 384-channel "recording" of the neurowaves?