Different results using different OS for an R package

[eridona]

Hello! We are trying to reproduce some analysis using different OS (Alma 8 and Ubuntu22.04) and find some differences in the results. We believe that the differences are due to BLAS/LAPACK libraries. We are using R-4.2.1 and using the Seurat package to run dimensionality reduction analysis (PCA). Both R point to libraries "libRblas.so" and "libRlapack.so". When checking for libraries dependencies we get different results on the two systems. Does that mean that we are using two different versions of the libraries on the two machines?

On Ubuntu: ldd /R/R-4-2/lib/R/lib/libRblas.so

linux-vdso.so.1 (0x00007ffd0c7a9000)
libm.so.6 => /usr/lib/x86_64-linux-gnu/libm.so.6 (0x00007f60428f4000)
libc.so.6 => /usr/lib/x86_64-linux-gnu/libc.so.6 (0x00007f60426cc000)
/lib64/ld-linux-x86-64.so.2 (0x00007f6042a0a000)

while ldd /R/R-4-2/lib/R/lib/libRlapack.so

linux-vdso.so.1 (0x00007ffde6969000)
libRblas.so => /R/R-4-2/lib/R/lib/libRblas.so (0x00007ff972831000)
libgfortran.so.5 => /usr/lib/x86_64-linux-gnu/libgfortran.so.5 (0x00007ff972556000)
libm.so.6 => /usr/lib/x86_64-linux-gnu/libm.so.6 (0x00007ff97246f000)
libc.so.6 => /usr/lib/x86_64-linux-gnu/libc.so.6 (0x00007ff972247000)
libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007ff9721fd000)
libgcc_s.so.1 => /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ff9721dd000)
/lib64/ld-linux-x86-64.so.2 (0x00007ff972a33000)

In Alma: ldd /R/R-4.2/lib64/R/lib/libRblas.so

linux-vdso.so.1 (0x00007ffdd33f6000)
libRblas.so => /public/flybrain/R/R-4.2/lib64/R/lib/libRblas.so (0x000014d43736a000)
libgfortran.so.5 => /public/gcc/9_1_0/lib64/libgfortran.so.5 (0x000014d436edf000)
libm.so.6 => /usr/lib64/libm.so.6 (0x000014d436b5d000)
libquadmath.so.0 => /public/gcc/9_1_0/lib64/libquadmath.so.0 (0x000014d436917000)
libgomp.so.1 => /public/gcc/9_1_0/lib64/libgomp.so.1 (0x000014d4366e1000)
libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x000014d4364c1000)
libc.so.6 => /usr/lib64/libc.so.6 (0x000014d4360fc000)
libgcc_s.so.1 => /public/gcc/9_1_0/lib/../lib64/libgcc_s.so.1 (0x000014d435ee4000)
/lib64/ld-linux-x86-64.so.2 (0x000014d437983000)
libdl.so.2 => /usr/lib64/libdl.so.2 (0x000014d435ce0000)

while ldd /R/R-4.2/lib64/R/lib/libRblas.so

linux-vdso.so.1 (0x00007ffc039e1000)
libgfortran.so.5 => /public/gcc/9_1_0/lib64/libgfortran.so.5 (0x000014b17690c000)
libm.so.6 => /usr/lib64/libm.so.6 (0x000014b17658a000)
libquadmath.so.0 => /public/gcc/9_1_0/lib64/libquadmath.so.0 (0x000014b176344000)
libgomp.so.1 => /public/gcc/9_1_0/lib64/libgomp.so.1 (0x000014b17610e000)
libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x000014b175eee000)
libc.so.6 => /usr/lib64/libc.so.6 (0x000014b175b29000)
libgcc_s.so.1 => /public/gcc/9_1_0/lib/../lib64/libgcc_s.so.1 (0x000014b175911000)
/lib64/ld-linux-x86-64.so.2 (0x000014b176fc2000)
libdl.so.2 => /usr/lib64/libdl.so.2 (0x000014b17570d000)

What would you advice to achieve the exact same results in both systems?

Thank you for your help!

Erika

[martin-frbg]

Looks like your Ubuntu installation is using a BLAS that does not use Fortran, probably one of the optimized implementations, likely OpenBLAS or ATLAS. On both systems, libRblas and libRlapack are probably softlinks to the actual library rather than real files. Looking at these should then tell you what libraries (and versions) are actually behind this. Possibly you can also use the update-alternatives command to query the current setup and change it (at least within the limits of what library choices are provided in either system)

[eridona]

Thanks! the libRblas.so is not a softlink, maybe it got written when compiling R from source?

[martin-frbg]

Possibly. Obviously in that case it depends on how R was built - as far as I know, the R sources contain a stripped-down version of Reference-LAPACK's BLAS and LAPACK but also support building with whatever BLAS and LAPACK are already installed on the system. Unfortunately I have no idea if it copies the system library to libRblas.so in the latter case, but I would think a link is more logical). Maybe the sessionInfo() command in R can provide additional information on the exact nature of the library ? (This here is probably not the best place to get help about R installation problems)

[eridona]

Do you think that using the same library version in different macines should give exactly the same results? Would you maybe be able to point me for two identical BLAS/LAPACK packages for Alma 8 and Ubuntu22.04?

[martin-frbg]

Maybe if you use the unoptimized BLAS and LAPACK, compiled with the exact same compiler version (and probably without any compiler options for optimization as well). As far as I know there are a few projects underway for developing a "reproducible BLAS", e.g. the ReproBLAS of Ahrens et al. at Berkeley. I am not aware of a corresponding LAPACK, but I am not an expert in this field. You could try to find packages of the same release version of Reference-LAPACK for both Alma8 and Ubuntu22, but either distribution may have added post-release patches to the source, and their precompiled packages will most likely have been built with whatever was their default compiler version at the time.