Sundials.jl
Sundials.jl copied to clipboard
SciML
External library support?
By default Sundials is not compiled with blas/lapack.
I changed build.ji to this:
`./configure --prefix=$prefix --enable-shared --with-blas=/usr/lib/libblas.so.3 --with-lapack=/usr/lib/liblapack.so.3`
It compiled all right, and generated cvode_lapack.o and cvode_lapack.o
Then I added CVLapackDense to cvode.jl
# header: /usr/local/include/cvode/cvode_lapack.h
@c Int32 CVLapackDense (Ptr{:None},:Cint) shlib
I naively thought that's all I needed to do to enable lapack.
However, when I tried to use Sundials.CVLapackDense(mem, neq) instead of Sundials.CVDense(mem, neq) and run Sundials.CVode, my program crashed with no reasons given.
Is there any way to enable lapack in Sundials?
We need to provide an easy way for libraries to link to Julia's BLAS/LAPACK. @tkelman Do we have a good way to do this yet?
What is your versioninfo() ? Which Linux distribution and blas package are you using?
Not all libraries are going to be flexible enough to handle an ILP64 Blas/Lapack, or one with renamed synbols. If Julia had used conventional LP64 Blas all along, then interoperating with existing libraries like this one would be easier.
Ubuntu 14.04
Julia Version 0.3.11 Commit 483dbf5 (2015-07-27 06:18 UTC) Platform Info: System: Linux (x86_64-linux-gnu) CPU: Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz WORD_SIZE: 64 BLAS: libblas.so.3 LAPACK: liblapack.so.3 LIBM: libopenlibm LLVM: libLLVM-3.3
My blas & lapack are atlas. Sundials requires f77 inplace to link blas.
@ViralBShah no we do not ship an LP64 blas in default binaries. It would be quite large and only of benefit to packages. If packages need an LP64 blas they can be responsible for obtaining one. Since this is a need shared by multiple packages we may want to centralize the bindeps handling for this in a single small package that other packages could then depend on.
@colinfang Where is that build of Julia from? If it was a source build, what was in your Make.user? What is LinAlg.BlasInt in that build?
I simply used the official version https://launchpad.net/~staticfloat/+archive/ubuntu/juliareleases
And after install, swapped the blas from openblas to atlas using update-alternatives.
Sundials still crashes if I use openblas + lapack3 (reference lapack).
We're recommending the PPA less over time, in favor of the generic linux tarballs.
Do the julia tests pass Base.runtests() after switching out blas and lapack? It's not a good idea to use a different blas than julia was built with.
The crash could be an incorrect ccall or almost anything. Run in julia-debug under gdb and try to get a backtrace.
Using atlas, it didn't pass linalg test as expected.
Using openblas + lapack3, it hangs after reaching parallel test, seems to be common due to some bug noted in issue I guess. But I think the blas part works all right.
I will see if I can gdb Sundials cvode.
It turned out IJulia ate my traceback.
So it is a symbol lookup error: libsundials_cvodes.so: undefined symbol: dcopy_
And I ldd libsundials_cvodes.so but didn't find any libblas.so link
linux-vdso.so.1 => (0x00007ffc63710000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1d20162000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1d1fd9d000)
/lib64/ld-linux-x86-64.so.2 (0x00007f1d206c0000)
You may need to change the --with-blas to use -L and -l linker flags, and/or patch sundials' build system to get the cvode library to link properly against blas and lapack. Specifying the full library file path may have worked for linking the c library but it's underlinked for dynamically loading in julia.
I don't know if I linked Sundials library correctly with blas. Based on the Sundials documentation the only place I need to change to enable blas is through .configure, and I did that. But I don't know how to verify if it is properly linked . (the only way I know is from ldd ...). Maybe I have to give up.
I'm expanding this topic from BLAS/LAPACK support to external library support and consolidating conversations with @tshort and @MartinOtter here.
Essentially, Sundials can support the following external libraries:
- BLAS
- LAPACK
- KLU
- SuperLU_MT
- PETSc
- hypre
- CUDA
- raja
BLAS is only needed because other libraries use it. If LAPACK, KLU, or SuperLU_MT are built with Sundials, then adding support for their linear solvers into the direct and common interfaces is very easy (it's essentially putting one more option in the if statements). CUDA, hypre, RAJA, and PETSc (+MPI) would require setting up and testing different N_Vector types which is a lot more work and a project itself (possible GSoC?). Thus for now, I would like to try and target the first 3 and leave the others for later.
KLU was reasonably easy to build. That should be easy to integrate in the library build script from this. @MartinOtter, did the libraries from SundialsKLUBuilder work with Modia?
Elliot has also made a builder for OpenBLAS, so that should be a good starting point for SuperLU_MT.
This thread is on LAPACK. https://github.com/JuliaDiffEq/SundialsBuilder/issues/6 We have Sundials building with OpenBLAS, which has LAPACK, but I haven't found out how to bind it to LAPACK. If someone is willing to play around with the build script, I'll update wrap_sundials.jl and the common interface wrapper and then it will default to LAPACK and be multithreaded (which will be a big upgrade!).
Just have to link to libopenblas.
I tried that here: https://github.com/JuliaDiffEq/SundialsBuilder/pull/11 . I am doing this remotely through Travis since I got rid of the VM I was using before, but Travis doesn't throw the build errors so I just gave up for now. It probably is just a simple step away from what I was trying. The other issue is that Sundials doesn't allow LAPACK with 64-bit, so I patched that out and that might be what the real build problem is. I don't know why they disable LAPACK when 64-bit.
https://github.com/JuliaDiffEq/SundialsBuilder/pull/9
Maybe use SuiteSparseBuilder too so that we can reuse things?
Yeah, it was mentioned in https://github.com/JuliaDiffEq/SundialsBuilder/issues/6 . It's a good idea, but doing so doesn't have a clear deliverable I haven't found the motivation to change it. Honestly I think it's a good first step to a Sundials parallelization GSoC 2019 (start by changing to SuiteSparseBuilder, then build with OpenMP/pthreads, add in SuperLUMT, and then get this all wrapped in Sundials.jl) so I'm leaving it open. But adding LAPACK should be a very easy change with a big impact on users, so I'd prefer to not have that one sit as long.
Hmm, looking through their lapack bindings, it is not easy to use it for 64-bit lapack in the build system. But it only seems to be dispatching to 4 lapack functions, where we could easily patch them.
Actually looks like about a dozen or so lapack calls. I think we could insert some shims into their lapack calling so that it can then call our 64-bit lapack with the mangled names and ILP64, if the effort is worthwhile.
If it's that much work then it can wait. Yingbo will have a JVODE (Julia version of Sundials) done in about a month which will of course use LAPACK and multithreading, so we're getting close to the point where Sundials is good to keep updated but is becoming more of a dev tool to benchmark against than an actual user tool. So if it's easy, yes. If not, don't.
Sundials 3.1.1 is also in BB and is compiled with KLU support. It was not compatible with the ilp64 blas and lapack (failed tests), but sundials 5.1 is, and it passes tests.
Sundials.jl probably needs to be updated for sundials 5.1.