Artifact sandbox broken in Test environment

[vchuravy]

On 1.8-rc1/1.9.0-DEV.1065, but fine on 1.7

@v1.8) pkg> generate SomePkg
  Generating  project SomePkg:
    SomePkg/Project.toml
    SomePkg/src/SomePkg.jl

shell> mkdir SomePkg/test

shell> touch SomePkg/test/runtests.jl

(@v1.8) pkg> activate SomePkg/test
  Activating new project at `~/SomePkg/test`

(test) pkg> add LLVM_jll
    Updating registry at `~/.julia/registries/CESMIX`
    Updating git-repo `https://github.com/Cesmix-mit/CESMIX`
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
    Updating registry at `~/.julia/registries/MolSim`
    Updating git-repo `https://github.com/JuliaMolSim/MolSim`
   Resolving package versions...
    Updating `~/SomePkg/test/Project.toml`
⌅ [86de99a1] + LLVM_jll v13.0.1+3
    Updating `~/SomePkg/test/Manifest.toml`
  [692b3bcd] + JLLWrappers v1.4.1
  [21216c6a] + Preferences v1.3.0
⌅ [86de99a1] + LLVM_jll v13.0.1+3
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [b77e0a4c] + InteractiveUtils
  [b27032c2] + LibCURL v0.6.3
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.8.0
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [fa267f1f] + TOML v1.0.0
  [a4e569a6] + Tar v1.10.0
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [deac9b47] + LibCURL_jll v7.83.1+1
  [29816b5a] + LibSSH2_jll v1.10.2+0
  [c8ffd9c3] + MbedTLS_jll v2.28.0+0
  [14a3606d] + MozillaCACerts_jll v2022.2.1
  [83775a58] + Zlib_jll v1.2.12+3
  [8f36deef] + libLLVM_jll v13.0.1+2
  [8e850ede] + nghttp2_jll v1.47.0+0
  [3f19e933] + p7zip_jll v17.4.0+0
        Info Packages marked with ⌅ have new versions available but cannot be upgraded. To see why use `status --outdated -m`

(test) pkg> activate SomePkg
  Activating project at `~/SomePkg`

(SomePkg) pkg> test
  No Changes to `~/SomePkg/Project.toml`
  No Changes to `~/SomePkg/Manifest.toml`
     Testing SomePkg
      Status `/tmp/jl_FcgnTz/Project.toml`
ERROR: LoadError: ArgumentError: Package TOML not found in current path.
- Run `import Pkg; Pkg.add("TOML")` to install the TOML package.
Stacktrace:
 [1] macro expansion
   @ ./loading.jl:1163 [inlined]
 [2] macro expansion
   @ ./lock.jl:223 [inlined]
 [3] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1144
 [4] include(fname::String)
   @ Base.MainInclude ./client.jl:476
 [5] top-level scope
   @ none:5
in expression starting at /home/vchuravy/.julia/packages/LLVM_jll/44Mtf/.pkg/select_artifacts.jl:3
ERROR: failed process: Process(`/usr/bin/julia -Cnative -J/usr/lib/julia/sys.so -g1 -O0 --color=no --history-file=no --startup-file=no --compiled-modules=yes --project=/tmp/jl_FcgnTz/Project.toml --eval 'append!(empty!(Base.DEPOT_PATH), ["/home/vchuravy/.julia", "/usr/local/share/julia", "/usr/share/julia"])
append!(empty!(Base.DL_LOAD_PATH), String[])

cd("/home/vchuravy/.julia/packages/LLVM_jll/44Mtf/.pkg")
include("/home/vchuravy/.julia/packages/LLVM_jll/44Mtf/.pkg/select_artifacts.jl")
' x86_64-linux-gnu-libgfortran5-cxx11-libstdcxx29-julia_version+1.8.0`, ProcessExited(1)) [1]

[staticfloat]

So it turns out that this may not be a Pkg bug but rather a Julia problem. Pointing julia-1.7 at a release-1.7 checkout of Pkg, the issue does not appear, but pointing julia-1.8 at a release-1.7 checkout of Pkg, the issue does appear.

Bisecting julia yields the following:

$ git bisect bad
a4e2cfb722dc3a4e8cf95179f03b425568b0df21 is the first bad commit
commit a4e2cfb722dc3a4e8cf95179f03b425568b0df21
Author: Valentin Churavy <[email protected]>
Date:   Fri Feb 11 18:59:56 2022 -0500

    Upgrade LLVM to 13.0.1

 deps/Versions.make              |  10 +-
 deps/checksums/clang            | 116 +++++-----
 deps/checksums/llvm             | 468 ++++++++++++++++++++--------------------
 deps/llvm.mk                    |   6 +-
 deps/llvm.version               |   4 +-
 stdlib/libLLVM_jll/Project.toml |   2 +-
 6 files changed, 301 insertions(+), 305 deletions(-)

Equal parts irony and tragedy, it appears that once again, our beloved Valentin has partaken in the debugging murder mystery, where he is simultaneously the victim and the perpetrator. Looking at the diff for a4e2cfb, it hardly seems possible that this could be the root issue. It's so innocuous! So perhaps there is some problem in the added JLL? Looking at the diff of the two libLLVM_jll versions is similarly unenlightening. And then, a bolt of inspiration strikes; it's not that libLLVM_jll has new code, it's that changing the version of libLLVM_jll allows new code. Indeed, the issue is that by upgrading the version of libLLVM_jll, we allow a new LLVM_jll, which has the new .pkg/select_artifacts.jl hook.

The whole idea that it's somehow Julia version dependent is a red herring; the version constraints on LLVM_jll simply restrict us to an older version of that package on Julia 1.7-, and that older version does not contain the select_artifacts.jl hook. Digging into the actual problem, it becomes evident that this is due to the gen_build_code() function clearing the entire LOAD_PATH, and the fact that during testing, we have no access to the standard library. Fix here: https://github.com/JuliaLang/Pkg.jl/pull/3167