GR backend interaction with system libraries

[EetuReijonen]

Hi, I have run into a weird issue when trying to use Plots.jl with GRBackend and Gurobi.jl (the optimization solver) with multiple workers.

The minimal working example is here:

using Plots
gr()
plot()

using Distributed
addprocs(4)
@everywhere begin
    using Gurobi
    Gurobi.Env()
end

This code will produce an error, which, according to my understanding, indicates that the Gurobi library/program cannot be found.

Plots.GRBackend()

4-element Vector{Int64}:
 2
 3
 4
 5

Set parameter Username
Academic license - for non-commercial use only - expires 2024-05-21
ERROR: On worker 2:
could not load library "libgurobi100.dylib"
dlopen(libgurobi100.dylib, 0x0001): tried: 'libgurobi100.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibgurobi100.dylib' (no such file), '/Applications/Julia-1.9.1.app/Contents/Resources/julia/lib/julia/libgurobi100.dylib' (no such file), '/Applications/Julia-1.9.1.app/Contents/Resources/julia/lib/julia/../libgurobi100.dylib' (no such file), '/Applications/Julia-1.9.1.app/Contents/Resources/julia/lib/libgurobi100.dylib' (no such file), '/usr/lib/libgurobi100.dylib' (no such file, not in dyld cache), 'libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/5902919ba30b97bafa388ca950a351ea32516d26/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/92b949e2f3a66439c69a8d334fc95810fbd9df9b/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/5b338c8fa90c05e6faea86e54d2996cca76cfbbe/lib/libgurobi100.dylib' (no such file), '/Applications/Julia-1.9.1.app/Contents/Resources/julia/lib/julia/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/d18fd9a0903c01ba831674a057a5ef7caf6aeec6/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/9410bad2635eda2239b4a72ba4316c4aa8f5b76e/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/95f332911a8dd5ab82620ba58c217193bd0d515d/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/10b2b258c07d7a76b2e3331f1ed70d8a8eb6d71c/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/9a76a401f82e0e3cafce618fb8d2d5c307ab2836/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/4aea97e4407a13e6d2ef7e5a05864ed1db642163/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/c65e07e3da4f1bf519bc432389dbbd61df320457/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/4ec62d729213a748d2300dd0832ebe8ed2292093/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/e6b9fb44029423f5cd69e0cbbff25abcc4b32a8f/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/b2108f561a8812e376eb80e71a24a3678a24d231/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/ada2a202928dd4cb2fc4bd18c4efa9d5455ec742/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/df3881e810714d6a09467fe85a6fde79385fe702/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/3b3d0bcaf14a9b239a4f4dc20ef7b9e63030a47e/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/abf161ac3d4df76ae74bbf5432b7e061b3876236/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/4260cf51a368d8e305a5de3669e32539e1e6cc72/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/fc7ba632b72ce7d852c1924aa2bbfe244a71c780/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/413111420faa4e2aeaa383c075eaa213402d939c/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/ca2831bf6edc5088aec5b329ea98364951d6cad0/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/3fe6bf926e57cc4be598151cd40832221de2e894/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/c325a23bc1f6521474cef5f634f18c8ab311bb02/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/0db9c3f6cf936a0da49e2ba954ba3e10bed6ad72/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/1a7e22e66b523d9cb884cf85c3ec065b5fb3e5c3/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/4a64372f135495d207e45abb25f29f9f66d46d7a/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/4609432e7098d8434a7a4c7876dd5b9e09b2a5e7/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/6095fcd268ea712c0f786f5ff1a45bf0eb7b005e/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/8dfccc7f1aa9d156f12a6e7db599014b805b11f2/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/b80d54484d815d1ace4cb475d53da633579d7d92/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/b450526929615030746974fd622effa333c2c87a/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/fb653fdc268e50c0be4391f6ca2f2fce2be7f15b/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/9a9f59eab237f7454fee1d6ab112a254032540b7/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/2107e7bc404f11b178cb9724cb371ef704995727/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/4b3b2d79556cc3aef6e3d8a234649cc85b91bb87/lib/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/dc1fe71ef939802ae7490350407bea0f47e5be5f/lib/QtConcurrent.framework/Versions/A/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/dc1fe71ef939802ae7490350407bea0f47e5be5f/lib/QtCore.framework/Versions/A/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/dc1fe71ef939802ae7490350407bea0f47e5be5f/lib/QtDBus.framework/Versions/A/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/dc1fe71ef939802ae7490350407bea0f47e5be5f/lib/QtGui.framework/Versions/A/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/dc1fe71ef939802ae7490350407bea0f47e5be5f/lib/QtNetwork.framework/Versions/A/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/dc1fe71ef939802ae7490350407bea0f47e5be5f/lib/QtOpenGL.framework/Versions/A/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/dc1fe71ef939802ae7490350407bea0f47e5be5f/lib/QtPrintSupport.framework/Versions/A/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/dc1fe71ef939802ae7490350407bea0f47e5be5f/lib/QtSql.framework/Versions/A/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/dc1fe71ef939802ae7490350407bea0f47e5be5f/lib/QtTest.framework/Versions/A/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/dc1fe71ef939802ae7490350407bea0f47e5be5f/lib/QtWidgets.framework/Versions/A/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/dc1fe71ef939802ae7490350407bea0f47e5be5f/lib/QtXml.framework/Versions/A/libgurobi100.dylib' (no such file), '/Users/eetureijonen/.julia/artifacts/f1af74bc0a21f699d4974bdceac10d4638e61e74/lib/libgurobi100.dylib' (no such file), '/Applications/Julia-1.9.1.app/Contents/Resources/julia/bin/../lib/julia/libgurobi100.dylib' (no such file), '/Applications/Julia-1.9.1.app/Contents/Resources/julia/bin/../lib/libgurobi100.dylib' (no such file), '/libgurobi100.dylib' (no such file)
Stacktrace:
  [1] GRBemptyenv
    @ ~/.julia/packages/Gurobi/yvF0V/src/gen100/libgrb_api.jl:813 [inlined]
  [2] _
    @ ~/.julia/packages/Gurobi/yvF0V/src/MOI_wrapper/MOI_wrapper.jl:108
  [3] Env
    @ ~/.julia/packages/Gurobi/yvF0V/src/MOI_wrapper/MOI_wrapper.jl:102
  [4] top-level scope
    @ ~/Desktop/GAMMA-OPT/Gogeta.jl/examples/neural_networks_parallel.jl:34
  [5] eval
    @ ./boot.jl:370
  [6] #invokelatest#2
    @ ./essentials.jl:816
  [7] invokelatest
    @ ./essentials.jl:813
  [8] #114
    @ /Applications/Julia-1.9.1.app/Contents/Resources/julia/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:301
  [9] run_work_thunk
    @ /Applications/Julia-1.9.1.app/Contents/Resources/julia/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:70
 [10] run_work_thunk
    @ /Applications/Julia-1.9.1.app/Contents/Resources/julia/share/julia/stdlib/v1.9/Distributed/src/process_messages.jl:79
 [11] #100
    @ ./task.jl:514

...and 3 more exceptions.

Stacktrace:
 [1] sync_end(c::Channel{Any})
   @ Base ./task.jl:445
 [2] macro expansion
   @ ./task.jl:477 [inlined]
 [3] remotecall_eval(m::Module, procs::Vector{Int64}, ex::Expr)
   @ Distributed /Applications/Julia-1.9.1.app/Contents/Resources/julia/share/julia/stdlib/v1.9/Distributed/src/macros.jl:219
 [4] top-level scope
   @ /Applications/Julia-1.9.1.app/Contents/Resources/julia/share/julia/stdlib/v1.9/Distributed/src/macros.jl:203

However, by using a different backend (e.g. PyPlot), the code works just fine:

using Plots
pyplot()
plot()

using Distributed
addprocs(4)
@everywhere begin
    using Gurobi
    Gurobi.Env()
end

Output:

Plots.PyPlotBackend()

4-element Vector{Int64}:
 2
 3
 4
 5

Set parameter Username
Academic license - for non-commercial use only - expires 2024-05-21
      From worker 5:    Set parameter Username
      From worker 4:    Set parameter Username
      From worker 4:    Academic license - for non-commercial use only - expires 2024-05-21
      From worker 3:    Set parameter Username
      From worker 3:    Academic license - for non-commercial use only - expires 2024-05-21
      From worker 2:    Set parameter Username
      From worker 2:    Academic license - for non-commercial use only - expires 2024-05-21
      From worker 5:    Academic license - for non-commercial use only - expires 2024-05-21

I'm using Julia version 1.9.1, Plot.jl v1.39.0, Gurobi.jl v1.2.1 on a Macbook M2 Air with macOS 14.2

Could this issue be caused by some kind of alteration to the system environment before it's passed to the workers? Or possibly a namespace conflict (Gurobi is also abbreviated GR in some contexts)?

[jheinen]

I have no idea, why this happens. I don't think, that there are name conflicts (GR vs GRB).

Could please find the missing library (find ~/.julia/artifacts -name libgurobi100.dylib) and try the following command:

otool -L <path from above find command>libgurobi100.dylib

You could also try to run the otool command within your Julia session.

[EetuReijonen]

For some reason I cannot find the Gurobi library in my artifacts folder (or any .dylib file with gurobi in the name).

eetureijonen@Eetus-MacBook-Air artifacts % find ~/.julia/artifacts -type f \( -name "*gurobi*" -a -name "*.dylib" \)

I found it in my /Library folder and this is the output from the otool-command.

eetureijonen@Eetus-MacBook-Air lib % otool -L /Library/gurobi1001/macos_universal2/lib/libgurobi100.dylib
/Library/gurobi1001/macos_universal2/lib/libgurobi100.dylib (architecture x86_64):
	/Library/gurobi1001/macos_universal2/lib/libgurobi100.dylib (compatibility version 10.0.0, current version 10.0.1)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1770.255.0)
	/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
	/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 1109.60.2)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 59754.60.13)
/Library/gurobi1001/macos_universal2/lib/libgurobi100.dylib (architecture arm64):
	/Library/gurobi1001/macos_universal2/lib/libgurobi100.dylib (compatibility version 10.0.0, current version 10.0.1)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
	/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1770.255.0)
	/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
	/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 1109.60.2)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 59754.60.13)

[jheinen]

/Library/gurobi1001/macos_universal2/lib is not in the dlopen search path, so you could try to explicitly set it before starting Julia:

export DYLD_LIBRARY_PATH="/Library/gurobi1001/macos_universal2/lib:${DYLD_LIBRARY_PATH}"
julia ...

[EetuReijonen]

Thank you, this seems to solve the acute problem for me.

eetureijonen@Eetus-MacBook-Air examples % export DYLD_LIBRARY_PATH="/Library/gurobi1001/macos_universal2/lib:${DYLD_LIBRARY_PATH}"
eetureijonen@Eetus-MacBook-Air examples % julia --project
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.9.1 (2023-06-07)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using Plots

julia> gr()
Plots.GRBackend()

julia> plot()

julia> using Distributed

julia> addprocs(4)
4-element Vector{Int64}:
 2
 3
 4
 5

julia> @everywhere begin
           using Gurobi
           Gurobi.Env()
       end
Set parameter Username
Academic license - for non-commercial use only - expires 2024-05-21
      From worker 2:    Set parameter Username
      From worker 3:    Set parameter Username
      From worker 3:    Academic license - for non-commercial use only - expires 2024-05-21
      From worker 2:    Academic license - for non-commercial use only - expires 2024-05-21
      From worker 5:    Set parameter Username
      From worker 4:    Set parameter Username
      From worker 5:    Academic license - for non-commercial use only - expires 2024-05-21
      From worker 4:    Academic license - for non-commercial use only - expires 2024-05-21

However, I think there is still something going on as other backends do not cause the same issue.