mpifileutils
mpifileutils copied to clipboard
dsync and probably dcp doesn't handle hardlinks
Hardlinks are not handled and causes errors on the destination files: source: /a/b hardlinked to /a/c dsync /a /target
[2020-10-31T05:44:52] [39] [/path/src/common/mfu_flist_copy.c:330] ERROR: Failed to change ownership on /target/c' lchown() (errno=2 No such file or directory) [2020-10-31T05:44:52] [39] [/path/src/common/mfu_flist_copy.c:356] ERROR: Failed to change permissions on
/target/c' chmod() (errno=2 No such file or directory)
And one more for utime
And /target/b and /target/c ends up as separate files, sometimes with one of them (c in the above case) having incorrect timestamps.
Currently running mpifileutils 0.10.1
The permission, utime errors might be seomthing unrelated, I see the same problem on non-hardlinked files. And yes, the files do exist afterwards.
For Lustre at least, it is possible to get a list of hard links for a file, up to a limit of 4KB (depends on length of filenames, but in the range of 10-50 reverse links). This covers over 99% of the cases for hard-linked files.
And of course I forgot to say that these are Lustre FS:es both of them.
On the other hand, rsync doesn't handle hard links unless given -H/--hard-links, so neither should dsync.
But such an option should be added.