LZFSE does not work on large dmg files -> consider bringing another working implementation upstream?

[AlexeyGy]

First of all, big thanks for maintaining the project!

the example devdiskimage.img as described in commit https://github.com/Lekensteyn/dmg2img/commit/a2d8dfb59dc81ce7c46bb384ff02a8a21ca15d72

➜  test-lzfse ./dmg2img/dmg2img DeveloperDiskImage.dmg devdiskimage.img -d -v   

dmg2img v1.6.5 (c) vu1tur ([email protected])

DeveloperDiskImage.dmg --> devdiskimage.img

Debug info will be written to dmg2img.log

reading property list, 4218 bytes from address 19871934 ...
partition 0: begin=174, size=1912, decoded=1284

decompressing:
opening partition 0 ...        [27] 100.00%  ok

Archive successfully decompressed as devdiskimage.img

works fine. It is ~20MB in size.

However with a much bigger file like IntelliJ 2020.2.4 (~670MB), there is an error:

~ wget https://download.jetbrains.com/idea/ideaIC-2020.2.4.dmg 
~ ./dmg2img/dmg2img ./ideaIC-2020.2.4.dmg ideaIC-2020.2.4.iso -d 

dmg2img v1.6.5 (c) vu1tur ([email protected])

./ideaIC-2020.2.4.dmg --> ideaIC-2020.2.4.iso

Debug info will be written to dmg2img.log
decompressing:
opening partition 0 ...             100.00%  ok
opening partition 1 ...             100.00%  ok
opening partition 2 ...             100.00%  ok
opening partition 3 ...             100.00%  ok
opening partition 4 ...               4.77%Truncated 1054586 to 1048576, data might be lost
decompression failed

and the extraction fails.

There is a fork out there by citruz where the extraction works for me (checked md5 against hdiutil). It only contains a single commit so should be easy to bring upstream.

In their implementation, they mention that "lzfse does not support chunked decompression". Hence, they decompress the whole block at once. This is different from the implementation here.

Would you perhaps consider integrating the solution by citruz? If you need any help, do not hesitate to ask 👍 .

The code I used to test:

wget https://download.jetbrains.com/idea/ideaIC-2020.2.4.dmg
# other repo https://github.com/citruz/dmg2img/
git clone https://github.com/Lekensteyn/dmg2img/
git clone https://github.com/lzfse/lzfse/

# compile and install lzfse
cd lzfse
make && sudo make install
cd ..

# compile and install dmg2img
cd dmg2img
make dmg2img HAVE_LZFSE=1 || true
cd ..

./dmg2img/dmg2img ./ideaIC-2020.2.4.dmg ideaIC-2020.2.4.iso -V -d
7z x ideaIC-2020.2.4.iso -oidea-extract

#### On Linux ####
# for file comparing, to also copy hidden files
shopt -s dotglob nullglob
mv idea-extract/IntelliJ\ IDEA\ CE/* idea-extract/
rmdir idea-extract/IntelliJ\ IDEA\ CE

find idea-extract -type f -exec md5sum {} + | sort -k 2 > linux.txt

#### on OSX only ####

wget https://download.jetbrains.com/idea/ideaIC-2020.2.4.dmg

hdiutil makehybrid -iso -joliet -o ideaIC-2020.2.4.iso ideaIC-2020.2.4.dmg
7z x ideaIC-2020.2.4.iso -oidea-extract
brew install md5sha1sum
find idea-extract -type f -exec md5sum {} + | sort -k 2 > osx.txt

#### Finally ####
diff linux.txt osx.txt