python-can icon indicating copy to clipboard operation
python-can copied to clipboard

Optional use of ISA-L for zlib decode/encode operations

Open belliriccardo opened this issue 1 year ago • 0 comments

Is your feature request related to a problem? Please describe.

I have an application that requires the decoding of very large .blf files, with hundreds of millions of can messages. I've modified the BLFReaderclass a lot, but I have observed that a very large speedup can be achieved very easily just by using a more optimized zlib decompression library, such as Intel's ISA-L (and the bindings provided by the isal python package) as a drop-in replacement.

Describe the solution you'd like

The zlib import statement inside can/io/blf.py could be simply changed from:

import zlib

to:

try:
    from isal import isal_zlib as zlib
except ImportError:
    import zlib

This can be so that the library can be listed as optional in the installation instructions, and the remaining code can remain unmodified. I don't know about making it mandatory and/or replacing it altogether, as I haven't tested the compress routine (it isn't used in my specific application), but it should be fine.

Also I really haven't checked if the zlib module is used anywhere else, but obviously this change can be "universal".

Additional context

This speedup works best when you need "skip" a lot of messages in the log (e.g. if you're only interested in can messages with a specific arbitration id and skip all of the others), and therefore need to sift trough this .blf file very quickly. I don't have any precise figures on the % speedup because it is very much context dependent, but it can be significant, especially on such very large files; also the package's installed size is around 540KB, so not too heavy.

belliriccardo avatar Jun 19 '24 06:06 belliriccardo