Maximilian Knespel

Add optimized file access that also works for non-seekable files

3

There are some use cases that want to create the index while downloading, so something like `wget | tee downloaded-file | pragzip --export-index downlaoded-file.gzindex`. I think this should be solvable...

Add compression for the index format

1

For files with very large compression ratios, the index might actually grow larger than the original file because the seek points are 32 KiB of uncompressed data for each chunk...

This might make https://github.com/mxmlnkn/ratarmount/issues/105 more performance. Implementing it purely on the ratarmount side would result in having to open one ParallelGzipReader for each compressed file. This can quickly lead to...

Untested ideas for performance improvements: - Use `hpx::thread`, `hpx::lock`, ... instead of `std::thread`. - [x] Use [rpmalloc](https://github.com/mjansson/rpmalloc) (showed ~10% speedup), I also tested [mimalloc](https://github.com/microsoft/mimalloc) and [jemalloc](https://github.com/jemalloc/jemalloc). This only needs to...

Add parallelized compression

2

The most difficult part is probably the build system and packaging because currently only the files necessaryfor decompression from zlib and ISA-l are built and packaged. Aside from that, simply...

Support for other specialized gzip formats similar to BGZF

2

Rapidgzip already supports BGZF files and also indexes created by `indexed_gzip`. Support for more obscure similar gzip file formats would be necessary because gzip files consisting only of multiple gzip...

Deflate blocks with Fixed Huffman codings are currently ignored because only three bits (final bit, block type = 0b01) can be checked and the final block bit check might also...

gztool has something like this: > windows are not loaded in memory unless they're needed, so the application memory footprint is fairly low I think it should be fairly easy...

It seems that seeking inside lz4 should be easily possible the same way as for gzip. The only difference is that the lz4 window size is [64 KiB](https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md) instead of...

Don't simply quit on bad CRC?

5

In order to read from faulty media it might be helpful to not just "crash" with an exception when a bad crc is encountered. I did simply quit because when...