rmlint
rmlint copied to clipboard
sahib
Something goes wrong when piping list of files to 'rmlint --hash', with the -a option used
This is funny, there's no problems when I either pass the list of files in the command line
>> rmlint --hash -a md5 file1 file2 file3
4e5010b82394c82473c84dbb63df2491 file1
c9ebe8afa4b1898ede7cb72a09375160 file2
9b68cf1980b1c63965f5dbfda248d95a file3
or if I don't use the -a option:
>> cat list | rmlint --hash
af6f4cf09f43b853c1544202b41d12607e3dcd9b0491b4eb868367c758c10dc1a1b1f2b52bf1b72adcdac9156706c1e20e5fd9311cbb7a0ddfcbf732af57d0e2 /home/panos/tbd/file1
b54ec8ec2bbab2e439183faf603e2a18e46496df70d6e16440fb8f4e9710b4c71edb6cfe146b2ca03f0af129b95bfc87a3dcb1ce66028a10794400f2b1bb8418 /home/panos/tbd/file2
8bcf7958331f48576294ba667a4e358db2842342c2d4d6e4a5aa16ac40d66e7e1d334b7829895b4b711bcb0b51f4fb57f0d9a807de2ab88988255a3ddcdf4b3b /home/panos/tbd/file3
But if the files are piped to rmlint AND the -a flag is used, things fall apart: Interestingly, the hashes are all produced norally. The error only seems to appear at the very end of the process (I notice this when hashing big files)
>> cat list | rmlint --hash -a md5
4e5010b82394c82473c84dbb63df2491 /home/panos/tbd/file1
WARNING: c9ebe8afa4b1898ede7cb72a09375160 /home/panos/tbd/file2
Can't open directory or file "metro": No such file or directory
9b68cf1980b1c63965f5dbfda248d95a /home/panos/tbd/file3
ERROR: Aborting due to a fatal error. (signal received: Segmentation fault)
ERROR: Please file a bug report (See rmlint -h)
>> cat list | rmlint --hash -a blake2b
af6f4cf09f43b853c1544202b41d12607e3dcd9b0491b4eb868367c758c10dc1a1b1f2b52bf1b72adcdac9156706c1e20e5fd9311cbb7a0ddfcbf732af57d0e2 /home/panos/tbd/file1
WARNING: b54ec8ec2bbab2e439183faf603e2a18e46496df70d6e16440fb8f4e9710b4c71edb6cfe146b2ca03f0af129b95bfc87a3dcb1ce66028a10794400f2b1bb8418 /home/panos/tbd/file2
Can't open directory or file "metro": No such file or directory
8bcf7958331f48576294ba667a4e358db2842342c2d4d6e4a5aa16ac40d66e7e1d334b7829895b4b711bcb0b51f4fb57f0d9a807de2ab88988255a3ddcdf4b3b /home/panos/tbd/file3
ERROR: Aborting due to a fatal error. (signal received: Segmentation fault)
ERROR: Please file a bug report (See rmlint -h)
The error doesn't always print in the same order, I guess since this runs in multiprocess this is just a case of a race condition for printing in the terminal.
When trying with a list of normal files I also get a .... memory dump?
WARNING: Can't open directory or file "": No such file or directory
*** Error in `rmlint': free(): invalid pointer: 0x00000000008214b0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7c503)[0x7f4aecccf503]
/lib64/libglib-2.0.so.0(g_free+0xe)[0x7f4aed79b7ee]
rmlint[0x40f60f]
rmlint[0x409b2d]
rmlint[0x40a3de]
rmlint[0x406d15]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f4aecc74b35]
rmlint[0x4068b9]
======= Memory map: ========
00400000-0044a000 r-xp 00000000 fd:00 403519004 /usr/bin/rmlint
00649000-0064a000 r--p 00049000 fd:00 403519004 /usr/bin/rmlint
0064a000-0064d000 rw-p 0004a000 fd:00 403519004 /usr/bin/rmlint
00812000-008ad000 rw-p 00000000 00:00 0 [heap]
7f4ad8000000-7f4ad8022000 rw-p 00000000 00:00 0
7f4ad8022000-7f4adc000000 ---p 00000000 00:00 0
7f4adc000000-7f4adc021000 rw-p 00000000 00:00 0
7f4adc021000-7f4ae0000000 ---p 00000000 00:00 0
7f4ae0000000-7f4ae0021000 rw-p 00000000 00:00 0
7f4ae0021000-7f4ae4000000 ---p 00000000 00:00 0
7f4ae4201000-7f4ae4202000 ---p 00000000 00:00 0
7f4ae4202000-7f4ae4a02000 rw-p 00000000 00:00 0 [stack:22383]
7f4ae4a02000-7f4ae4a03000 ---p 00000000 00:00 0
7f4ae4a03000-7f4ae5203000 rw-p 00000000 00:00 0 [stack:22382]
7f4ae5203000-7f4aeb72c000 r--p 00000000 fd:00 201338910 /usr/lib/locale/locale-archive
7f4aeb72c000-7f4aeb742000 r-xp 00000000 fd:03 321061195 /some/other/user/who/cantbebotheredtousevenvslikeahumanbeeingandhastousefucking/miniconda2/pkgs/libgcc-5.2.0-0/lib/libgcc_s.so.1
7f4aeb742000-7f4aeb941000 ---p 00016000 fd:03 321061195 /some/other/user/who/cantbebotheredtousevenvslikeahumanbeeingandhastousefucking/miniconda2/pkgs/libgcc-5.2.0-0/lib/libgcc_s.so.1
7f4aeb941000-7f4aeb942000 rw-p 00015000 fd:03 321061195 /some/other/user/who/cantbebotheredtousevenvslikeahumanbeeingandhastousefucking/miniconda2/pkgs/libgcc-5.2.0-0/lib/libgcc_s.so.1
7f4aeb942000-7f4aeb943000 rw-p 00074000 fd:03 321061195 /some/other/user/who/cantbebotheredtousevenvslikeahumanbeeingandhastousefucking/miniconda2/pkgs/libgcc-5.2.0-0/lib/libgcc_s.so.1
7f4aeb943000-7f4aeb983000 r-xp 00000000 fd:00 167814482 /usr/lib64/libmount.so.1.1.0
7f4aeb983000-7f4aebb83000 ---p 00040000 fd:00 167814482 /usr/lib64/libmount.so.1.1.0
7f4aebb83000-7f4aebb84000 r--p 00040000 fd:00 167814482 /usr/lib64/libmount.so.1.1.0
7f4aebb84000-7f4aebb85000 rw-p 00041000 fd:00 167814482 /usr/lib64/libmount.so.1.1.0
7f4aebb85000-7f4aebb86000 rw-p 00000000 00:00 0
7f4aebb86000-7f4aebb9c000 r-xp 00000000 fd:00 168025257 /usr/lib64/libresolv-2.17.so
7f4aebb9c000-7f4aebd9c000 ---p 00016000 fd:00 168025257 /usr/lib64/libresolv-2.17.so
7f4aebd9c000-7f4aebd9d000 r--p 00016000 fd:00 168025257 /usr/lib64/libresolv-2.17.so
7f4aebd9d000-7f4aebd9e000 rw-p 00017000 fd:00 168025257 /usr/lib64/libresolv-2.17.so
7f4aebd9e000-7f4aebda0000 rw-p 00000000 00:00 0
7f4aebda0000-7f4aebdc4000 r-xp 00000000 fd:00 167774005 /usr/lib64/libselinux.so.1
7f4aebdc4000-7f4aebfc3000 ---p 00024000 fd:00 167774005 /usr/lib64/libselinux.so.1
7f4aebfc3000-7f4aebfc4000 r--p 00023000 fd:00 167774005 /usr/lib64/libselinux.so.1
7f4aebfc4000-7f4aebfc5000 rw-p 00024000 fd:00 167774005 /usr/lib64/libselinux.so.1
7f4aebfc5000-7f4aebfc7000 rw-p 00000000 00:00 0
7f4aebfc7000-7f4aebfdc000 r-xp 00000000 fd:00 167774007 /usr/lib64/libz.so.1.2.7
7f4aebfdc000-7f4aec1db000 ---p 00015000 fd:00 167774007 /usr/lib64/libz.so.1.2.7
7f4aec1db000-7f4aec1dc000 r--p 00014000 fd:00 167774007 /usr/lib64/libz.so.1.2.7
7f4aec1dc000-7f4aec1dd000 rw-p 00015000 fd:00 167774007 /usr/lib64/libz.so.1.2.7
7f4aec1dd000-7f4aec23d000 r-xp 00000000 fd:00 167773996 /usr/lib64/libpcre.so.1.2.0
7f4aec23d000-7f4aec43c000 ---p 00060000 fd:00 167773996 /usr/lib64/libpcre.so.1.2.0
7f4aec43c000-7f4aec43d000 r--p 0005f000 fd:00 167773996 /usr/lib64/libpcre.so.1.2.0
7f4aec43d000-7f4aec43e000 rw-p 00060000 fd:00 167773996 /usr/lib64/libpcre.so.1.2.0
7f4aec43e000-7f4aec440000 r-xp 00000000 fd:00 168025235 /usr/lib64/libdl-2.17.so
7f4aec440000-7f4aec640000 ---p 00002000 fd:00 168025235 /usr/lib64/libdl-2.17.so
7f4aec640000-7f4aec641000 r--p 00002000 fd:00 168025235 /usr/lib64/libdl-2.17.so
7f4aec641000-7f4aec642000 rw-p 00003000 fd:00 168025235 /usr/lib64/libdl-2.17.so
7f4aec642000-7f4aec645000 r-xp 00000000 fd:00 167814488 /usr/lib64/libgmodule-2.0.so.0.5600.1
7f4aec645000-7f4aec844000 ---p 00003000 fd:00 167814488 /usr/lib64/libgmodule-2.0.so.0.5600.1
7f4aec844000-7f4aec845000 r--p 00002000 fd:00 167814488 /usr/lib64/libgmodule-2.0.so.0.5600.1
7f4aec845000-7f4aec846000 rw-p 00003000 fd:00 167814488 /usr/lib64/libgmodule-2.0.so.0.5600.1
7f4aec846000-7f4aec84d000 r-xp 00000000 fd:00 167774782 /usr/lib64/libffi.so.6.0.1
7f4aec84d000-7f4aeca4c000 ---p 00007000 fd:00 167774782 /usr/lib64/libffi.so.6.0.1
7f4aeca4c000-7f4aeca4d000 r--p 00006000 fd:00 167774782 /usr/lib64/libffi.so.6.0.1
7f4aeca4d000-7f4aeca4e000 rw-p 00007000 fd:00 167774782 /usr/lib64/libffi.so.6.0.1
7f4aeca4e000-7f4aeca52000 r-xp 00000000 fd:00 167814469 /usr/lib64/libuuid.so.1.3.0
7f4aeca52000-7f4aecc51000 ---p 00004000 fd:00 167814469 /usr/lib64/libuuid.so.1.3.0
7f4aecc51000-7f4aecc52000 r--p 00003000 fd:00 167814469 /usr/lib64/libuuid.so.1.3.0
7f4aecc52000-7f4aecc53000 rw-p 00004000 fd:00 167814469 /usr/lib64/libuuid.so.1.3.0
7f4aecc53000-7f4aece09000 r-xp 00000000 fd:00 168025229 /usr/lib64/libc-2.17.so
7f4aece09000-7f4aed009000 ---p 001b6000 fd:00 168025229 /usr/lib64/libc-2.17.so
7f4aed009000-7f4aed00d000 r--p 001b6000 fd:00 168025229 /usr/lib64/libc-2.17.so
7f4aed00d000-7f4aed00f000 rw-p 001ba000 fd:00 168025229 /usr/lib64/libc-2.17.so
7f4aed00f000-7f4aed014000 rw-p 00000000 00:00 0
7f4aed014000-7f4aed02b000 r-xp 00000000 fd:00 168025255 /usr/lib64/libpthread-2.17.so
7f4aed02b000-7f4aed22a000 ---p 00017000 fd:00 168025255 /usr/lib64/libpthread-2.17.soERROR: Aborting due to a fatal error. (signal received: Aborted)
ERROR: Please file a bug report (See rmlint -h)
>> rmlint --version
version 2.9.0 compiled: Sep 18 2019 at [18:39:14] "Odd Olm" (rev unknown)
compiled with: +mounts +nonstripped +fiemap +sha512 +bigfiles +intl +replay +xattr +btrfs-support
rmlint was written by Christopher <sahib> Pahl and Daniel <SeeSpotRun> Thomas.
The code at https://github.com/sahib/rmlint is licensed under the terms of the GPLv3.
>> uname -a
Linux kindaprivatenodename 3.10.0-514.10.2.el7.x86_64 #1 SMP Fri Mar 3 00:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
it's a bit old, I think it's whatever I could grab from yum
I can't reproduce the behavior on my machine. Please try the latest commit from the develop branch (see the docs on how to compile that).
If that persists, I gonna have you run this command (after installing valgrind):
$ cat list | G_DEBUG=gc-friendly G_SLICE=always-malloc valgrind --leak-check=full --show-possibly-lost=no ./rmlint --hash -a blake2b
This command should be run with your self compiled binary since it contains debug information.
I tried following the steps of the guide but unfortunately run into errors:
librmlint.a(cfg.o): In function `rm_cfg_add_path':
cfg.c:(.text+0x508): undefined reference to `g_canonicalize_filename'
collect2: error: ld returned 1 exit status
scons: *** [rmlint] Error 1
Admittedly this might be a quirk of the system, it's probably missing a ton of important library upgrades, in fear of not messing up what works. So l guess we can leave it at that for now and Ill let you know if I run into the problem once the machine is upgraded.
I tried following the steps of the guide but unfortunately run into errors: [...]
Your GLib is tool old. You need >= : 2.58 currently with the develop branch.
Admittedly this might be a quirk of the system, it's probably missing a ton of important library upgrades, in fear of not messing up what works. So l guess we can leave it at that for now and Ill let you know if I run into the problem once the machine is upgraded.
Okay. I close this issue for now. Please re-open when there's new input.
The issue mentioned in the title is real, and I was able to reproduce it. The array --hash reads stdin paths into is missing a terminating NULL, so it exits if you're lucky and crashes if you're not. It doesn't always happen but it's not hard to reproduce if you try a longer list of paths (or in the OP's case, the -a option). I wrote a fix a while ago and I plan to make it available soon.
