Update of SNOPT version and use precompiled binary for libsnopt7

[DGCaprace]

Newer versions of SNOPT ship with precompiled libraries (static or dynamic), and not the source code anymore. This PR makes it possible for SNOPT.jl to either link to the provided dynamic library, or to include the content of the static library and repackage it for use with Julia.

The user can still copy the SNOPT source code files to ./deps/src, like before. However, if the source code is not available, the user can copy the binary to ./deps/src. With a small modification of the Makefile (instructions provided in the README), the make command will be able to link to the dynamic library (libsnopt7.so/libsnopt7.dylib) or include the content of the static archive (libsnopt7.a).

[DGCaprace]

What currently works

The modified Makefile works and enables building Snopt.jl. Note that, in the case of dynamic linking, some dependencies may or may not be found at execution time. This may require to export a LD_LIBRARY_PATH on Linux or DYLD_LIBRARY_PATH on MacOS with the list of path to the dependencies. You can check what are the dependencies by running (Linux): ldd libsnopt.so (MacOS): otool -L libsnopt.dylib

What currently doesn't work

Even when all the dependencies are found, so far, I experienced issues with the precompiled library which throws a nice segfault.

On MacOS

Using either the dynamic or static library compiled for MacOS with GCC, I currently get:

(@v1.6) pkg> test Snopt
     Testing Snopt
      Status `/private/var/folders/xf/tyzqbz1n4vj7ycrllg2c8cvw0000gq/T/jl_2Cayhl/Project.toml`
  [0e9dc826] Snopt v0.2.2 `~/Documents/BYU/devel/Julia/Snopt.jl`
  [2f01184e] SparseArrays `@stdlib/SparseArrays`
  [8dfed614] Test `@stdlib/Test`
      Status `/private/var/folders/xf/tyzqbz1n4vj7ycrllg2c8cvw0000gq/T/jl_2Cayhl/Manifest.toml`
  [0e9dc826] Snopt v0.2.2 `~/Documents/BYU/devel/Julia/Snopt.jl`
  [2a0f44e3] Base64 `@stdlib/Base64`
  [b77e0a4c] InteractiveUtils `@stdlib/InteractiveUtils`
  [8f399da3] Libdl `@stdlib/Libdl`
  [37e2e46d] LinearAlgebra `@stdlib/LinearAlgebra`
  [56ddb016] Logging `@stdlib/Logging`
  [d6f4376e] Markdown `@stdlib/Markdown`
  [9a3f8284] Random `@stdlib/Random`
  [9e88b42a] Serialization `@stdlib/Serialization`
  [2f01184e] SparseArrays `@stdlib/SparseArrays`
  [8dfed614] Test `@stdlib/Test`
     Testing Running tests...

signal (11): Segmentation fault: 11
in expression starting at /Users/dg/Documents/BYU/devel/Julia/Snopt.jl/test/runtests.jl:8
snseti_ at /Users/dg/Documents/BYU/devel/Julia/Snopt.jl/deps/src/v7.7.7/libsnopt7.dylib (unknown line)
setmemory at /Users/dg/Documents/BYU/devel/Julia/Snopt.jl/src/Snopt.jl:356
#snopta#6 at /Users/dg/Documents/BYU/devel/Julia/Snopt.jl/src/Snopt.jl:576
snopta##kw at /Users/dg/Documents/BYU/devel/Julia/Snopt.jl/src/Snopt.jl:493 [inlined]
#snopta#5 at /Users/dg/Documents/BYU/devel/Julia/Snopt.jl/src/Snopt.jl:460 [inlined]
snopta at /Users/dg/Documents/BYU/devel/Julia/Snopt.jl/src/Snopt.jl:458
snopta at /Users/dg/Documents/BYU/devel/Julia/Snopt.jl/src/Snopt.jl:458
unknown function (ip: 0x10d0c0d4f)
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
do_call at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
eval_body at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
eval_body at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
eval_body at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_interpret_toplevel_thunk at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_toplevel_eval_flex at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_toplevel_eval_flex at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_toplevel_eval_in at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
eval at ./boot.jl:360 [inlined]
include_string at ./loading.jl:1094
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
_include at ./loading.jl:1148
include at ./client.jl:444
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
do_call at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
eval_body at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_interpret_toplevel_thunk at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_toplevel_eval_flex at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_toplevel_eval_flex at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_toplevel_eval_in at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
eval at ./boot.jl:360 [inlined]
exec_options at ./client.jl:261
_start at ./client.jl:485
jfptr__start_32022.clone_1 at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
true_main at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
repl_entrypoint at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
Allocations: 2281450 (Pool: 2280374; Big: 1076); GC: 3
ERROR: Package Snopt errored during testing (received signal: 11)

This might be due to how the libs were compiled. I am thinking that the compilation may have used a -march option incompatible with the architecture I am testing things on. That being said, when I try to determine for which architecture the file was compiled:

objdump -f libsnopt7.a 

I can only see x86_64 which should not be an issue.

On Linux

Using the static lib compiled for Linux, this time the compilation throws an error:

gfortran -shared -O2 -fPIC   fileio.o -L./v7.7.7/   -Wl,--whole-archive /zhome/dcaprace/myJulia/Snopt.jl/deps/src/v7.7.7/libsnopt7.a -Wl,--no-whole-archive   -o libsnopt.so 
/bin/ld: /zhome/dcaprace/myJulia/Snopt.jl/deps/src/v7.7.7/libsnopt7.a(snopta.o): relocation R_X86_64_PC32 against symbol `snkera_' can not be used when making a shared object; recompile with -fPIC
/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
make: *** [libsnopt.so] Error 1

The error seems to indicate that libsnopt7.a was not compiled with the -fPIC option. This seems to be backed-up by the information here.

Using the dynamic lib compiled for Linux, it compiles (of course) but then when I run, I get:

(@v1.6) pkg> test Snopt
     Testing Snopt
      Status `/tmp/jl_LA8aUr/Project.toml`
  [0e9dc826] Snopt v0.2.2 `~/myJulia/v1.6/Snopt.jl`
  [2f01184e] SparseArrays `@stdlib/SparseArrays`
  [8dfed614] Test `@stdlib/Test`
      Status `/tmp/jl_LA8aUr/Manifest.toml`
  [0e9dc826] Snopt v0.2.2 `~/myJulia/v1.6/Snopt.jl`
  [2a0f44e3] Base64 `@stdlib/Base64`
  [b77e0a4c] InteractiveUtils `@stdlib/InteractiveUtils`
  [8f399da3] Libdl `@stdlib/Libdl`
  [37e2e46d] LinearAlgebra `@stdlib/LinearAlgebra`
  [56ddb016] Logging `@stdlib/Logging`
  [d6f4376e] Markdown `@stdlib/Markdown`
  [9a3f8284] Random `@stdlib/Random`
  [9e88b42a] Serialization `@stdlib/Serialization`
  [2f01184e] SparseArrays `@stdlib/SparseArrays`
  [8dfed614] Test `@stdlib/Test`
     Testing Running tests...

signal (11): Segmentation fault
in expression starting at /home/dcaprace/myJulia/v1.6/Snopt.jl/test/runtests.jl:8
snseti_ at /home/dcaprace/myJulia/v1.6/Snopt.jl/deps/src/v7.7.7/libsnopt7.so (unknown line)
setmemory at /home/dcaprace/myJulia/v1.6/Snopt.jl/src/Snopt.jl:351
#snopta#6 at /home/dcaprace/myJulia/v1.6/Snopt.jl/src/Snopt.jl:571
snopta##kw at /home/dcaprace/myJulia/v1.6/Snopt.jl/src/Snopt.jl:488 [inlined]
#snopta#5 at /home/dcaprace/myJulia/v1.6/Snopt.jl/src/Snopt.jl:455 [inlined]
snopta at /home/dcaprace/myJulia/v1.6/Snopt.jl/src/Snopt.jl:453
snopta at /home/dcaprace/myJulia/v1.6/Snopt.jl/src/Snopt.jl:453
unknown function (ip: 0x7f100c731f80)
jl_apply at /tmp/julia-1.6.1/src/julia.h:1703 [inlined]
do_call at /tmp/julia-1.6.1/src/interpreter.c:115
eval_value at /tmp/julia-1.6.1/src/interpreter.c:204
eval_stmt_value at /tmp/julia-1.6.1/src/interpreter.c:155 [inlined]
eval_body at /tmp/julia-1.6.1/src/interpreter.c:562
eval_body at /tmp/julia-1.6.1/src/interpreter.c:490
eval_body at /tmp/julia-1.6.1/src/interpreter.c:490
jl_interpret_toplevel_thunk at /tmp/julia-1.6.1/src/interpreter.c:670
jl_toplevel_eval_flex at /tmp/julia-1.6.1/src/toplevel.c:877
jl_toplevel_eval_flex at /tmp/julia-1.6.1/src/toplevel.c:825
jl_toplevel_eval_in at /tmp/julia-1.6.1/src/toplevel.c:929
eval at ./boot.jl:360 [inlined]
include_string at ./loading.jl:1094
_include at ./loading.jl:1148
include at ./client.jl:444
jl_apply at /tmp/julia-1.6.1/src/julia.h:1703 [inlined]
do_call at /tmp/julia-1.6.1/src/interpreter.c:115
eval_value at /tmp/julia-1.6.1/src/interpreter.c:204
eval_stmt_value at /tmp/julia-1.6.1/src/interpreter.c:155 [inlined]
eval_body at /tmp/julia-1.6.1/src/interpreter.c:562
jl_interpret_toplevel_thunk at /tmp/julia-1.6.1/src/interpreter.c:670
jl_toplevel_eval_flex at /tmp/julia-1.6.1/src/toplevel.c:877
jl_toplevel_eval_flex at /tmp/julia-1.6.1/src/toplevel.c:825
jl_toplevel_eval_in at /tmp/julia-1.6.1/src/toplevel.c:929
eval at ./boot.jl:360 [inlined]
exec_options at ./client.jl:261
_start at ./client.jl:485
jfptr__start_42540 at /zapps7/julia/1.6.1/gcc-10.2.0/lib/julia/sys.so (unknown line)
jl_apply at /tmp/julia-1.6.1/src/julia.h:1703 [inlined]
true_main at /tmp/julia-1.6.1/src/jlapi.c:560
repl_entrypoint at /tmp/julia-1.6.1/src/jlapi.c:702
main at /tmp/julia-1.6.1/cli/loader_exe.c:51
__libc_start_main at /lib64/libc.so.6 (unknown line)
_start at /zapps7/julia/1.6.1/gcc-10.2.0/bin/julia (unknown line)
Allocations: 2289326 (Pool: 2288239; Big: 1087); GC: 3

One thing we can glean here is that this Julia (BYU supercomputer module Julia/v1.6) seems to be build on top of GCC10 and libc.so.6, whereas the libsnopt7 bin we have depends on libgfortran.so.4 which seems to be from a previous GCC version (e.g. GCC7). Not sure if that can cause compatibility issues.

[DGCaprace]

FYI: @tylercritchfield and I tried this on his MAC and we observed the same behavior.

[andrewning]

@DGCaprace should I merge this or hold? I know we were never able to get the supplied precompiled libraries to work.

[DGCaprace]

The code in this branch is backward compatible, i.e. the default behavior is still to compile from the user-provided source code. So yes it could be merged, unless you want to leave the PR open until someone figures out the issue with the binaries.