rsync segfaults when openssl fips is enabled

[sshedi]

Hi All,

System details:

root@ph3dev [ ~ ]# rsync --version
rsync  version 3.2.4  protocol version 31
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, symlinks, symtimes, hardlinks, hardlink-specials,
    hardlink-symlinks, IPv6, atimes, batchfiles, inplace, append, no ACLs,
    xattrs, optional protect-args, iconv, prealloc, stop-at, no crtimes
Optimizations:
    SIMD-roll, no asm-roll, openssl-crypto, no asm-MD5
Checksum list:
    xxh128 xxh3 xxh64 (xxhash) md5 md4 none
Compress list:
    zstd lz4 zlibx zlib none

OS:
Photon OS 3.0 with openssl fips enabled

I'm using rsync like below:

rsync -arXHpvxog <src> <dest>

While doing so, if any file in src or dest location has a xattr value which is more than 32 bytes long, rsync segfaults.

root@ph3dev [ ~ ]# getfattr -d -m '' -- /opt/x
getfattr: Removing leading '/' from absolute path names
# file: opt/x
user.bla1="1234567890abcdef1234567890abcdef1" --------> 33 bytes long

After examining the coredump file, this happens at; https://github.com/WayneD/rsync/blob/2f9b963abaa52e44891180fe6c0d1c2219f6686d/xattrs.c#L277

Here md5 is used by default and md5 usage is prohibited in fips mode.

Is there any way to workaround this problem? Also, why not use xxhash for this operation like it's used by default during --checksum option.

cc: @tapakund