dissect.target icon indicating copy to clipboard operation
dissect.target copied to clipboard

Add support to zstd compress vmtar file and replace zstandard with backport.zstd

Open william-billaud opened this issue 2 months ago • 1 comments

  • Add support to zstd compressed vmtar files (and XZ, as fsutil.open_decompress is now used to manage decompress)
  • Use VmtarFilesystem inside esxi._os, instead of manual tar uncompress
  • Add the backport.zstd dependency. In Python 3.14 zstd as been added to standard library (pep-0784), author as backported the lib.
    • Using this lib, we do not need to fully read data in memory when uncompressing zstd, which is more convenient.
  • Add an UncompressionError, mainly raised when required dependency is missing. This allows logging when a file couldn't be decompressed due to an error/missing deps
  • Initially, using zstandard I encounter uncompression Errors of few zstd file, but this error is not present using backport.zstd (and files from "faulty" archive are present when mounting the disk).
  • Add test regarding a zstd compressed vmtar file.

Some note regarding somme "issues" I found into the code

  • Perhaps zstandard usage should be fully replaced by backport.zstd/zstd in a dedicated issue
  • Perhaps a decompression error should be generated whenever a compressed stream is detected using open_decompress, but the required library is not present (or when decompression has failed). This would warn the user of something abnormal without causing a crash.

I can provide an esxi 8 and esxi 9 disk (30Go, less than 1Go compressed) if needed for testing purposes.

close #1296

william-billaud avatar Oct 29 '25 16:10 william-billaud

Regarding zstandard replacement it should be good. For the plocate plugins it is a bit more complicated as it need to read frame by frame, which is only supported for byte-like objects.

zstandard is still required due to dependencies in dissect.btrfs in dissect.squashfs

william-billaud avatar Nov 26 '25 16:11 william-billaud

I updated the plocate code to actually more accurately reflect how it should be parsed, instead of "brute forcing" blocks which it did before.

Schamper avatar Dec 04 '25 12:12 Schamper

Codecov Report

:x: Patch coverage is 66.66667% with 11 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 81.00%. Comparing base (34d04f2) to head (27bfa0f). :warning: Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
dissect/target/plugins/os/unix/esxi/_os.py 20.00% 4 Missing :warning:
dissect/target/tools/dump.py 40.00% 3 Missing :warning:
dissect/target/plugins/os/unix/log/journal.py 60.00% 2 Missing :warning:
dissect/target/helpers/fsutil.py 75.00% 1 Missing :warning:
dissect/target/plugins/os/unix/locate/plocate.py 92.85% 1 Missing :warning:
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1380      +/-   ##
==========================================
- Coverage   81.00%   81.00%   -0.01%     
==========================================
  Files         376      376              
  Lines       33660    33649      -11     
==========================================
- Hits        27266    27256      -10     
+ Misses       6394     6393       -1     
Flag Coverage Δ
unittests 81.00% <66.66%> (-0.01%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov[bot] avatar Dec 04 '25 12:12 codecov[bot]

CodSpeed Performance Report

Merging #1380 will not alter performance

Comparing william-billaud:esxi9_visor (27bfa0f) with main (34d04f2)

Summary

✅ 9 untouched

codspeed-hq[bot] avatar Dec 04 '25 12:12 codspeed-hq[bot]