mksquashfs speed drop when using lz4 high compression

[fmartinsons]

Hello, I was using mksquashfs 4.3 in an embedded environment (distro built using yocto) for a long time and I recently update to 4.4 (lz4 version 1.9.2)

I noticed a strong loss of speed when creating the squashfs, below is a reproduction using /usr as a source directory (I always used the proc and mem restrictions because my target have some sensible processus that don't like being too spoiled from resources)

4.3:

root@tap-0000C99F:~# mksquashfs -version
mksquashfs version 4.3-git (2014/09/12)
copyright (C) 2014 Phillip Lougher <[email protected]>

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2,
or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.
root@tap-0000C99F:~#
root@tap-0000C99F:~# du -sh /usr/
214.2M	/usr/
root@tap-0000C99F:~#
root@tap-0000C99F:~# time mksquashfs /usr/  test.squashfs -processors 1 -mem 16M -noappend -comp lz4 -Xhc
Parallel mksquashfs: Using 1 processor
Creating 4.0 filesystem on test.squashfs, block size 131072.
[========================================================================================================================================================================|] 7525/7525 100%

Exportable Squashfs 4.0 filesystem, lz4 compressed, data block size 131072
	compressed data, compressed metadata, compressed fragments, compressed xattrs
	duplicates are removed
Filesystem size 85656.21 Kbytes (83.65 Mbytes)
	36.28% of uncompressed filesystem size (236074.62 Kbytes)
Inode table size 92815 bytes (90.64 Kbytes)
	38.23% of uncompressed inode table size (242795 bytes)
Directory table size 93154 bytes (90.97 Kbytes)
	48.76% of uncompressed directory table size (191057 bytes)
Number of duplicate files found 412
Number of inodes 7282
Number of files 6273
Number of fragments 520
Number of symbolic links  292
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 717
Number of ids (unique uids + gids) 3
Number of uids 2
	<edited>
Number of gids 3
	<edited>

real	0m30.071s
user	0m11.044s
sys	0m0.962s

4.4:

root@tap-0000C99F:~# mksquashfs -version
mksquashfs version 4.4 (2019/08/29)
copyright (C) 2019 Phillip Lougher <[email protected]>

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2,
or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.
root@tap-0000C99F:~#
root@tap-0000C99F:~# du -sh /usr/
214.2M	/usr/
root@tap-0000C99F:~#
root@tap-0000C99F:~# time mksquashfs /usr/  test.squashfs -processors 1 -mem 16M -noappend -comp lz4 -Xhc
Parallel mksquashfs: Using 1 processor
Creating 4.0 filesystem on test.squashfs, block size 131072.
[================================================================================================================================================================================|] 7525/7525 100%

Exportable Squashfs 4.0 filesystem, lz4 compressed, data block size 131072
	compressed data, compressed metadata, compressed fragments,
	compressed xattrs, compressed ids
	duplicates are removed
Filesystem size 85355.88 Kbytes (83.36 Mbytes)
	36.15% of uncompressed filesystem size (236094.99 Kbytes)
Inode table size 92793 bytes (90.62 Kbytes)
	38.22% of uncompressed inode table size (242795 bytes)
Directory table size 92924 bytes (90.75 Kbytes)
	48.64% of uncompressed directory table size (191057 bytes)
Number of duplicate files found 412
Number of inodes 7282
Number of files 6273
Number of fragments 520
Number of symbolic links  292
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 717
Number of ids (unique uids + gids) 3
Number of uids 2
	<edited>
Number of gids 3
	<edited>

real	1m48.139s
user	0m42.668s
sys	0m0.523s

So 4 times factor !!

I dig a little and found https://github.com/lz4/lz4/issues/845 , so I decided to upgrade lz4 to 1.9.4, I had still the same speed loss. Then, I updated mksquashfs to 4.5.1 and again , the speed remained the same.

I continued my search and found that the culprit was this commit, after reverting it , I get back to the original speed I had with mksquashfs 4.3.

To better cornered the issue I also did the following:

• mksquashfs 4.3 with lz4 1.9.2: no speed loss
• mksquashfs 4.4 without c4f943ae6454955e3a5a76b8ff2c369d289d3068 and lz4 1.9.2: no speed loss
• mksquashfs 4.5.1 without c4f943ae6454955e3a5a76b8ff2c369d289d3068 and lz4 1.9.4: no speed loss

Any chance that you can find what is the problem here ? If you find that lz4 is finally the culprit, I'm ready to fill a bug in their repo too. Moreover, I'm available to make more tests on my setup if you want.

Thank you