-T LZVN doesn't work

[lucianmarin]

afsctool -T LZVN -c /path/to/random/file
/path/to/random/file: unsupported compression type 2 bytes
Unable to compress file (already compressed).

I think LZVN support is enabled (cmake LZVN repo then cmake afsctool), but it doesn't seem to work on High Sierra.

[RJVB]

afsctool -T LZVN -c /path/to/random/file

Two things seem to be going on here:

/path/to/random/file: unsupported compression type 2 bytes

This message indicates that LZVN support has in fact not been enabled.

Unable to compress file (already compressed).

And this suggests that your file was already compressed. But it can be an artefact because of LZVN support not having been enabled. What does /usr/bin/ls -lO /path/to/random/file print?

I think LZVN support is enabled (cmake LZVN repo then cmake afsctool), but it doesn't seem to work on High Sierra.

Did you install LZVN? Just building it isn't enough, the headers and library must be available in a location where cmake can find them. Cmake should report whether LZVN support is being enabled.

I've made a few small changes to afsctool so that it gives clearer error messages when LZVN support has not been enabled. Please pull afsctool, rebuild and try to compress your random file again. Then, install LZVN and re-cmake and re-build afsctool .

[lucianmarin]

make install LZVN

Install the project...
/usr/local/Cellar/cmake/3.13.2/bin/cmake -P cmake_install.cmake
-- Install configuration: ""
-- Up-to-date: /usr/local/lib/libFastCompression.1.0.dylib
-- Up-to-date: /usr/local/lib/libFastCompression.1.dylib
-- Up-to-date: /usr/local/lib/libFastCompression.dylib
-- Up-to-date: /usr/local/lib/libFastCompression.a
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: /usr/local/lib/libFastCompression.a(liblzvn.c.o) has no symbols
-- Up-to-date: /usr/local/bin/lzvn
-- Up-to-date: /usr/local/include/lzvn.h
-- Up-to-date: /usr/local/include/FastCompression.h

cmake -Wno-dev .. afsctool (after last git pull)

-- LZVN_HEADER,LIB: /usr/local/include/FastCompression.h /usr/local/lib/libFastCompression.dylib
-- Enabling LZVN support
-- Setting afsctool version to: 1.6.9. (1.6.8.6-127-g731aeaa)

Later edit: seems to work when I do cmake .. instead of cmake -Wno-dev .. and make install instead of make install/fast for both projects.

afsctool -v afsctool
File is HFS+/APFS compressed.
Compression type: LZVN in resource fork (8)
Compression savings: 71.0%

[lucianmarin]

Can you make LZVN the default compression format if support is enabled?

afsctool -c afsctool
afsctool -v afsctool
File is HFS+/APFS compressed.
Compression type: ZLIB in resource fork (4)
Compression savings: 73.6%

I also get lzvn compression failed on chunk #0 on some files if they were previously ZLIB compressed then decompressed then compressed with LZVN. ZLIB works on those files, but LZVN doesn't. They are too tiny for LZVN compression. lzvn infile outfile produces a 0 bytes outfile.

[RJVB]

I could but I won’t.

[gingerbeardman]

LZVN works for me on High Sierra, so you'll need to check that you are compiling afsctool correctly.

I use:

afsctool -c -T LZVN "$@"

I'm using afsctool 1.6.9. (1.6.8.6-125-ge269dbf) as I've not built a new version in a while.

[RJVB]

I also get lzvn compression failed on chunk #0 on some files if they were previously ZLIB compressed then decompressed then compressed with LZVN. ZLIB works on those files, but LZVN doesn't. They are too tiny for LZVN compression. lzvn infile outfile produces a 0 bytes outfile.

I didn't notice this part. I'd need to have one of those files to do some testing, but if lzvn fails itself there probably isn't much I can do about that.

I don't really get why you'd want to recompress files with LZVN though. In every test I did it compresses less well than (higher levels of) ZLIB. It may decompress faster, but do you really notice that IRL? OS X has a rather agressive file cache, and to the best of my knowledge this comes after the file system (i.e. it caches data after "HFS decompression").

[gingerbeardman]

FWIW I use LZVN because macOS App Store uses LZVN, and I mainly compress apps so I want to keep things the same as those apps.

[RJVB]

Whatever floats your boat :)

I tend to recompress apps to make them smaller, not to do things the way Apple do them. Apple also makes money from selling disk space so they have a vested interest in not minimising the space occupied by the software they sell ;)

[gingerbeardman]

I read that here the other day, and I countered by saying that Apple ship their OS and apps from the Mac App Store with HFS+ compression turned on, so I don't really see the reasoning?

My reason is not to "do it like Apple" but so that my scripts and workflow do not undo what Apple have done and then redo with the same. Also, LZVN is more efficient in terms of speed and CPU usage, so I don't notice when it's working in the background (compared to ZLIB that makes my fans go crazy). But, yes, whatever works for each of us.

Also, try FSNotes for an app that gives this:

/Applications/FSNotes.app/Contents/Resources/de.lproj/InfoPlist.strings: lzvn compression failed on chunk #0

[lucianmarin]

Also, try FSNotes for an app that gives this:

/Applications/FSNotes.app/Contents/Resources/de.lproj/InfoPlist.strings: lzvn compression failed on chunk #0

I can confirm this on multiple apps. LZVN utility produces 0 bytes output on the same files.

[RJVB]

Apple ship their OS and apps from the Mac App Store with HFS+ compression turned on

But not with the most efficient kind of compression.

lzvn compression failed on chunk #0

I can reproduce that. It's not overly surprising given that the input file is only 2 bytes large... It looks like there is an undocumented 8 byte lower limit to LZVN compression, under which it simply fails.

I pushed updates to lzvn and afsctool in order to ignore files < 8 bytes (and to improve verbose output a bit, the error I mentioned elsewhere that I wanted to fix).

[gingerbeardman]

Thanks René!

[gingerbeardman]

~~define for LZVN_MINIMUM_COMPRESSABLE_SIZE is missing in latest commit~~ make sure you pull, build and install the latest LZVN

[malthe]

Now we just need ARM-support for LZVN :-)

[RJVB]

Now we just need ARM-support for LZVN :-)

In the library I presume, or is there (also) an endianness issue in my own code?

[malthe]

In the library yeah – there's lots of assembly code in there.

[RJVB]

I'm going to compare that implementation with the generic-C implementation in liblzfse. Turns out there's one in there... If performance is close enough to the one from the assembly code I'm just going to use a single dependency.

[RJVB]

@malthe - Can you check the lzfse branch please (be sure to initialise the submodule!)? A priori it should provide LZVN and LZFSE support on both Apple's architectures now.

[malthe]

@RJVB works perfectly.

[RJVB]

@RJVB works perfectly.

That was fast! :) Both LZVN and LZFSE compression?

[malthe]

It takes quite a while, but this is an 8GB run:

Number of HFS+/APFS compressed files: 140960
Total number of files: 145251
Total number of folders: 20516
Total number of items (number of files + number of folders): 165767
Folder size (uncompressed; reported size by Mac OS 10.6+ Finder): 7840604172 bytes / 8.2 GB (gigabytes, base-10)
Folder size (compressed): 3696919434 bytes / 3.45 GiB
Compression savings: 52.8% over 140960 of 145251 files
Approximate total folder size (files + file overhead + folder overhead): 3704770889 bytes / 3.45 GiB

(Both LZVN and LZFSE tested.)

[RJVB]

You could have tested with a single file ;)

For reference, LZVN compression of a 128Mb file takes about 3.5s on my 2nd gen. i7 using the lzfse library. That's maybe 10% slower than with the assembly implementation. That surprises me a little bit, the C implementation almost looks like assembly, with lots of cases and gotos so it wouldn't surprise me if the assembly implementation actually came from an older version of that same code. We'll see if I get flamed for a performance regression from users on Intel hardware!

[gingerbeardman]

@malthe off-topic, slightly, but regarding your (old) blog post: https://maltheborch.com/#using-built-in-transparent-compression-on-macos ditto is in-built tool that can do this (HFS+ in-place compression) to a degree, and Apple do it for all system files in an install and all apps installed/updated using the Mac App Store. Just an FYI

[malthe]

@gingerbeardman thanks – I was planning to update that post once this is all merged in. Will try and see if I can come up with a command-line for ditto.

[gingerbeardman]

@malthe from man it's ditto --hfsCompression src dest

[malthe]

@gingerbeardman doesn't work if src is the same as dest though:

<src> and <dest> are identical (not copied).

[gingerbeardman]

ah yes, you have to do a dance through a temporary file

[RJVB]

FWIW, it's always been a pet theory of mine that Apple don't provide a tool for in-place compression (and even less so, transparent automatic FS-level compression) because they also sell storage space.

You may remember that it was long possible (and somewhat common usage) to clean off all the useless translation data (which often comes with copies of image resources) and indeed even the OS X installer allowed you to select to install only the languages you had use for (which would then be "completed" during subsequent updates which didn't have that option). App bundle signing has made that impossible.

[gingerbeardman]

I've heard that before, but my counter is that they apply this to the system and Mac App Store so they're doing more than nothing.

I do think they should enable it by default but there are so many issues that would raise.

Apple storage is pretty reasonable price these days.

[gingerbeardman]

Oh, and I use CleanMyMac to trim language files which still works well.