Add support to zstd compress vmtar file and replace zstandard with backport.zstd
- Add support to zstd compressed vmtar files (and XZ, as fsutil.open_decompress is now used to manage decompress)
- Use VmtarFilesystem inside esxi._os, instead of manual tar uncompress
- Add the backport.zstd dependency. In Python 3.14 zstd as been added to standard library (pep-0784), author as backported the lib.
- Using this lib, we do not need to fully read data in memory when uncompressing zstd, which is more convenient.
- Add an UncompressionError, mainly raised when required dependency is missing. This allows logging when a file couldn't be decompressed due to an error/missing deps
- Initially, using zstandard I encounter uncompression Errors of few zstd file, but this error is not present using backport.zstd (and files from "faulty" archive are present when mounting the disk).
- Add test regarding a zstd compressed vmtar file.
Some note regarding somme "issues" I found into the code
- Perhaps zstandard usage should be fully replaced by backport.zstd/zstd in a dedicated issue
- Perhaps a decompression error should be generated whenever a compressed stream is detected using
open_decompress, but the required library is not present (or when decompression has failed). This would warn the user of something abnormal without causing a crash.
I can provide an esxi 8 and esxi 9 disk (30Go, less than 1Go compressed) if needed for testing purposes.
close #1296
Regarding zstandard replacement it should be good. For the plocate plugins it is a bit more complicated as it need to read frame by frame, which is only supported for byte-like objects.
zstandard is still required due to dependencies in dissect.btrfs in dissect.squashfs
I updated the plocate code to actually more accurately reflect how it should be parsed, instead of "brute forcing" blocks which it did before.
Codecov Report
:x: Patch coverage is 66.66667% with 11 lines in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 81.00%. Comparing base (34d04f2) to head (27bfa0f).
:warning: Report is 1 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@
## main #1380 +/- ##
==========================================
- Coverage 81.00% 81.00% -0.01%
==========================================
Files 376 376
Lines 33660 33649 -11
==========================================
- Hits 27266 27256 -10
+ Misses 6394 6393 -1
| Flag | Coverage Δ | |
|---|---|---|
| unittests | 81.00% <66.66%> (-0.01%) |
:arrow_down: |
Flags with carried forward coverage won't be shown. Click here to find out more.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
:rocket: New features to boost your workflow:
- :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
CodSpeed Performance Report
Merging #1380 will not alter performance
Comparing william-billaud:esxi9_visor (27bfa0f) with main (34d04f2)
Summary
✅ 9 untouched