Realloc() causes OpenJPH slowdown on Windows OS

[palemieux]

The default HTJ2K compressor implementation uses the OpenJPH library. The mem_outfile class provided by OpenJPH. mem_outfile dynamically allocates memory using realloc(), which is extremely slow on Windows compared to Linux, resulting in the HTJ2K compressor being 10 times slower on Windows than on Linux.

The solution is to implement a class similar to mem_outfile that does not uses realloc().

[aous72]

Hi Pierre,

Thank you for looking at this issue, and solving it. These are my two cents from me.

1. The reason we use realloc is because we do not know the file size at the start. If we can estimate a rough upper bound of the file size, then we can use void open(size_t initial_size = 65536, bool clear_mem = false); setting the initial_size to this upper bound. For example, we can use width x height x num_components x bit_depth / 8 x 1.25; the 1.25 is just 25% overhead, which is a lot. Then, we are very sure that realloc is never called, unless we have some weird image, because this bound means that the compressed file is larger than the uncompressed data. In general we expect the losslessly compressed file to be 1/3 to 1/2 of the uncompressed data.
2. Control how fast the re-allocated memory grow; it is now 25% with each realloc().
3. It might be useful to reuse the mem_outfile object. The mem_outfile object can be closed (not deleted), and then opened again; all the allocated will be available for the new file, and it can also grow further if needed. This way, the start might be slow, but eventually things get as fast as can be -- I have to say I did not test this option thoroughly.

Cheers, Aous