lzma-rs icon indicating copy to clipboard operation
lzma-rs copied to clipboard

Implement the xz Delta filter (filter 3)

Open terrence2 opened this issue 1 year ago • 1 comments

Pull Request Overview

This pull request adds a delta compression filter and fixes multi-filter decompression; this appears to be the first test case making use of multiple filters.

Terrain height data tends to have lots of periodic repeating patterns, so GeoTiffs compressed with GDAL's liblzma support tend to have lots of tiles that make use of delta filtering. I need a native rust implementation of lzma decompression for use in OpenFA, so that I can port it to run on the web.

Testing Strategy

This pull request was tested by...

  • [x] Added relevant end-to-end tests (an .xz file with a delta filter).
  • [x] Integrated into OpenFA's GeoTiff parser and demonstrated equivalent function to liblzma.
  • [x] Ran cargo bench to verify there were no performance regressions.

Supporting Documentation and References

Code for the delta filter was adapted from the C code in xz-file-format.txt.

terrence2 avatar Jun 02 '23 02:06 terrence2

Great feedback! I think the new diff is much cleaner and clearer than the previous.

Since I was touching decomp_big_file anyway, I took the liberty of replacing read_all_file with the new fs::read builtin while I was there.

Please give it another look, at your convenience.

terrence2 avatar Nov 11 '23 15:11 terrence2