introlab
[Feature Request] Improve Data Compression
We have added RVL compression for depth images. However, in order to further reduce the size of the database, it is still necessary to compress other data. For point clouds and meshes, one possible solution is to use Google's Draco library.
Previously, other data was compressed using zlib. But I found its performance to be a bit poor. Even if only a small number of local features and descriptors are compressed, it will take more than ten milliseconds or more. Therefore, for intra-machine communication, I usually set Z_NO_COMPRESSION to avoid actual compression and use zero copy. From my experience with rosbag and ZFS, using lz4 compression should be a better choice. It sacrifices some compression, but is much faster.
Here other data (that are not images) that are currently compressed in the database: https://github.com/introlab/rtabmap/blob/f8b1d5065618b481cccfd777bb086744c0440962/corelib/src/resources/DatabaseSchema.sql.in#L37 https://github.com/introlab/rtabmap/blob/f8b1d5065618b481cccfd777bb086744c0440962/corelib/src/resources/DatabaseSchema.sql.in#L40-L42
These are point cloud data, so Draco library could be indeed used for that. For the compressed data under Admin table: https://github.com/introlab/rtabmap/blob/f8b1d5065618b481cccfd777bb086744c0440962/corelib/src/resources/DatabaseSchema.sql.in#L122-L130 as they are not meant to be used online, but only loaded/saved when rtabmap starts or closes, I would focus more on the ones above first. Note also that depending of the features used, a large part of the database can be from uncompressed descriptors.
Replacing zlib by lz4 may be a reasonable solution too. Ideally, if we could detect the compression format, we could add an option to compress general data as lz4 instead, while being backward compatible with databases with compressed data done with zlib. It looks like lz4 has a header, while zlib has its own, which means we could probably do something similar than you did for rvl versus png format.
That benchmark table is pretty useful: https://github.com/lz4/lz4?tab=readme-ov-file#benchmarks
