Document "shared depot" setup tips

[staticfloat]

I've received the same question a few times now about the proper way to set up a Julia installation such that it includes an extra entry on the DEPOT_PATH for the purposes of a "shared depot", e.g. lab-provided packages, global preference defaults, etc....

I know there are a few options for setting this up, (JULIA_DEPOT_PATH in the environment, pushing onto DEPOT_PATH in startup.jl, etc...) but most of the users who have asked are doing something like running a computer lab, and they don't want each user to have to modify their own startup.jl, or modify their .bash_profile to pick these things up. It would be nice if there were some way we could modify something in the Julia distribution itself that caused the DEPOT_PATH to be altered at startup.

Right now, I'm thinking the best way to do this is to modify the Julia installation to have the following put into etc/julia/startup.jl:

push!(Base.DEPOT_PATH, "/path/to/shared/environment")

Note that this will not work if the user starts up Julia with --startup-file=no, but I consider that a feature, not a bug. Is this the best thing we should suggest that folks do in this case, or is there a better idea?

[omlins]

I think the following brought up in #3061, would still be good to dicuss further (@staticfloat, any further comments?):

Me:

@staticfloat , @vchuravy : how about having in the loadpath a default location where the default preferences are looked up - analogue to @stdlib? It would be nice if we could be 100% sure that preferences defined in the Julia installation by cluster administrators are taken always into account if not explicitly and intentionally overwritten by the user- even in cases where the Pkg manager wants to create a "clean" environment etc.

@staticfloat :

This can be done by having users define a default JULIA_DEPOT_PATH that includes a system-wide depot. I don't suggest using @stdlib as this, since @stdlib has kind of a special meaning; it doesn't act like a true "depot". Let's open a new issue about the best way for system administrators to provide defaults (preferences, packages, etc...) for their users: https://github.com/JuliaLang/Pkg.jl/issues/3067

Me:

I don't suggest using https://github.com/stdlib as this, since https://github.com/stdlib has kind of a special meaning; it doesn't act like a true "depot".

My suggestion is not to abuse @stdlib for preferences, but to create in Pkg.jl an analogue to @stdlib for preferences, e.g. @stdprefs, which points to the standard location of preferences in the Julia installation.

@vchuravy :

Please add it as a comment to https://github.com/JuliaLang/Pkg.jl/issues/3067, I don't have any strong opinions, but we need to build community consensus around it.

[omlins]

@giordano ?

[vchuravy]

Also note that Elliot is currently on vacation

[einzigsue]

HI @omlins and @staticfloat,

Please, it would be super useful.

I read the Julia documentation about JULIA_DEPOT_PATH [https://docs.julialang.org/en/v1/manual/environment-variables/#JULIA_DEPOT_PATH] and DEPOT_PATH [https://docs.julialang.org/en/v1/base/constants/#Base.DEPOT_PATH] but I don't think it works for me like a stack of environment Pkg.jl would search through for a package.

For example when length(DEPOT_PATH)=3 it only reads through the first path, see details below.

Did I miss anything?

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.7.2 (2022-02-06)
 _/ |\__'_|_|_|\__'_|  |  HEAD/bf53498635 (fork: 461 commits, 337 days)
|__/                   |

julia> versioninfo()
Julia Version 1.7.2
Commit bf53498635 (2022-02-06 15:21 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin21.2.0)
  CPU: Apple M1
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, cyclone)
Environment:
  JULIA_DEPOT_PATH = /Users/Sue/Julia/tenv1:/Users/Sue/Julia/tenv2:/Users/Sue/Julia/tenv3

julia> DEPOT_PATH
3-element Vector{String}:
 "/Users/Sue/Julia/tenv1"
 "/Users/Sue/Julia/tenv2"
 "/Users/Sue/Julia/tenv3"

(v1.7) pkg> st
      Status `/Users/sue/Julia/tenv1/environments/v1.7/Project.toml`
  [179af706] CFTime v0.1.2

julia> using CFTime

julia> using CSV
 │ Package CSV not found, but a package named CSV is available from a registry. 
 │ Install package?
 │   (v1.7) pkg> add CSV 
 └ (y/n) [y]: n
ERROR: ArgumentError: Package CSV not found in current path:
- Run `import Pkg; Pkg.add("CSV")` to install the CSV package.

Stacktrace:
 [1] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:967

julia> using NLopt
 │ Package NLopt not found, but a package named NLopt is available from a
 │ registry. 
 │ Install package?
 │   (v1.7) pkg> add NLopt 
 └ (y/n) [y]: n
ERROR: ArgumentError: Package NLopt not found in current path:
- Run `import Pkg; Pkg.add("NLopt")` to install the NLopt package.

Stacktrace:
 [1] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:967

julia> popfirst!(DEPOT_PATH)
"/Users/Sue/Julia/tenv1"

julia> using CSV
[ Info: Precompiling CSV [336ed68f-0bac-5ca0-87d4-7b16caf5d00b]

julia> pathof(CSV)
"/Users/Sue/Julia/tenv2/packages/CSV/jFiCn/src/CSV.jl"

julia> DEPOT_PATH
2-element Vector{String}:
 "/Users/Sue/Julia/tenv2"
 "/Users/Sue/Julia/tenv3"

julia> using NLopt
 │ Package NLopt not found, but a package named NLopt is available from a registry. 
 │ Install package?
 │   (v1.7) pkg> add NLopt 
 └ (y/n) [y]: n
ERROR: ArgumentError: Package NLopt not found in current path:
- Run `import Pkg; Pkg.add("NLopt")` to install the NLopt package.

Stacktrace:
 [1] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:967

julia> popfirst!(DEPOT_PATH)
"/Users/Sue/Julia/tenv2"

julia> using NLopt
[ Info: Precompiling NLopt [76087f3c-5699-56af-9a33-bf431cd00edd]
...

julia> pathof(NLopt)
"/Users/Sue/Julia/tenv3/packages/NLopt/OIUOZ/src/NLopt.jl"

julia> 

[GregPlowman]

but I don't think it works for me like a stack of environment Pkg.jl would search through for a package.

For example when length(DEPOT_PATH)=3 it only reads through the first path, see details below.

Did I miss anything?

I also expected stacked environments to work as you described, but also observe that it seems only the first entry in DEPOT_PATH is searched.