undefined behaviour due to BLAS prototypes missing "hidden" variables required by Fortran

[conradsnicta]

The prototypes for Fortran BLAS and LAPACK functions appear to be incomplete in SuperLU. This can lead to undefined behaviour with recent releases of gcc/gfortran due to more aggressive compiler optimisations.

According to gfortran passing conventions (followed by other fortran compilers as well), every char* argument also has an associated so-called "hidden" argument which specifies the number of characters. The type of the "hidden" argument may vary across compilers and/or compiler versions. The position of each "hidden" argument is also compiler dependent, though seems to be typically tacked onto the end of the function definition. https://gcc.gnu.org/onlinedocs/gfortran/Argument-passing-conventions.html

As an example in SuperLU, the current prototype for dgemm in
https://github.com/xiaoyeli/superlu/blob/master/SRC/slu_ddefs.h
is defined as: extern int dgemm_(const char*, const char*, const int*, const int*, const int*, const double*, const double*, const int*, const double*, const int*, const double*, double*, const int*);

But it probably should be: extern int dgemm_(const char*, const char*, const int*, const int*, const int*, const double*, const double*, const int*, const double*, const int*, const double*, double*, const int*, fort_len, fort_len);

where fort_len is typically a 32 bit unsigned int on 32 bit platforms, and 64 bit unsigned int on 64 bit platforms (eg. size_t). This does not apply to gcc versions <= 7, where it is always int.

Until recently, avoiding the use of these fort_len arguments appeared to work, but new versions of gcc/gfortran are more strict, leading to stack overwrites (and hence crashes) when these arguments are not used.

The same bug affects a lot of other software in C and C++ that uses BLAS / LAPACK:

• https://gitlab.com/conradsnicta/armadillo-code/issues/123
• https://github.com/RcppCore/RcppArmadillo/issues/254
• https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90329
• https://gcc.gnu.org/ml/gcc-patches/2019-05/msg00915.html
• https://github.com/Reference-LAPACK/lapack/issues/339
• https://developer.r-project.org/Blog/public/2019/05/15/gfortran-issues-with-lapack/