Add a way to rewrite source file paths for re-located depots on the fly?

[oschulz]

When re-locating a whole Julia depot, most packages behave pretty well nowadays, with a few exceptions. This opens interesting possibilities for deploying Julia, especially to central environments where users have to deal with limited-size home directories (typically coupled with being on network/cluster file systems that don't like dealing with large numbers of files). But when re-locating depots that contain precompiled package images, source file lookup is incorrect, since the source file locations are "baked into" the precomilation output.

One recent use case is this: I've created https://github.com/JuliaHEP/juliahep_juliahep-base_img, which builds a Docker-image that is hosted on registry.cern.ch. It's mainly intended for use in particle physics, but is actually not that HEP-specific. The container image contains a lot of common packages and dependencies and their precompiled files (via /opt/julia-1.11/local/share/julia). It also sets $JULIA_PKG_PRESERVE_TIERED_INSTALLED=true. So when in a container instance, and adding packages, existing precompiled dependencies in the image will be used as much as possible, saving the user network homedir file space, inode-count, and time.

The folks that run the unpacked.cern.ch CVMFS repository have kindly agreed to host the container on CVMFS (in unpacked form). So users can, for example, just run apptainer shell /cvmfs/unpacked.cern.ch/registry.cern.ch/juliahep/juliahep-base\:latest on any system in the world that mounts CVMFS and that's it. CVMFS is a powerful read-only distribute software distribution system developed at CERN.

One cool thing about Julia though, is that external dependencies are provided down to basically libc-level. So it's possible to run the Julia from the unpacked image on CVMFS directly, bare-metal, without any container runtime (via a little included wrapper script), on basically any modern Linux that has CVMFS mounted:

$/cvmfs/unpacked.cern.ch/registry.cern.ch/juliahep/juliahep-base\:latest/unpacked/bin/julia

julia> DEPOT_PATH
3-element Vector{String}:
 "/home/username/.julia-juliahep-base"
 "/cvmfs/unpacked.cern.ch/.flat/b" ⋯ 68 bytes ⋯ "pt/julia-1.11/local/share/julia"
 "/cvmfs/unpacked.cern.ch/.flat/b" ⋯ 62 bytes ⋯ "833c/opt/julia-1.11/share/julia"

An advantage of running the unpacked image directly, without Apptainer (besides less complexity) is that one doesn't have to use tricks to run the VSCode-server-component inside of the Apptainer instance, for VSCode to be able to read/show files in the depot included in the container image.

But the annoying thing is that source file (when users want to dig through where stuff comes from, etc.) doesn't work, since it returns the original source file paths, not the relocated ones:

$/cvmfs/unpacked.cern.ch/registry.cern.ch/juliahep/juliahep-base\:latest/unpacked/bin/julia --project=@hep-base

julia> using Unitful; @which ustrip(5)
ustrip(x::Number)
     @ Unitful /opt/julia-1.11/local/share/julia/packages/Unitful/HTQlp/src/utils.jl:57

If we had source-file-lookup "re-routing" capability that one could hook into, things would be pretty much perfect. (when users want to dig through where stuff comes from, etc.)

CVMFS is available on quite a number of large systems now (NERSC, etc.), but I'm sure there are other scenarios where deployment of "pre-built depots" could be very useful.

According to @KristofferC we already rewrite source paths on the fy for stdlibs (from their original build location paths). He suggested I open an issue in this regard - could this be extended to moved depot packages in general?

[Keno]

I think it would be reasonable for Julia to persistently store only depot-relative (or potentially loaded-package-relative) paths and reconstruct them on demand. That would address this case. In general, I think Julia base should take a more active role in managing its source code, although I was planning to punt this until the 1.13 JuliaLowering integration.

[oschulz]

Thanks Keno! It would be awesome if this could happen in the 1.13 cycle. Would be too late for 1.12 anyway, right (since we're already at beta4)?

[Keno]

1.12 feature freeze happened months ago.

[KristofferC]

In this case, it's a bit unclear whether giving a non-existing path counts as a bug or not so 🤷

[oschulz]

In this case, it's a bit unclear whether giving a non-existing path counts as a bug or not so

I guess that depends on whether depot relocation is considered a supported feature in v1.12. ;-)

But I'll be more than happy if it might happen in v1.13, of course - I didn't assume changes like this could still happen in v1.12.

[giordano]

Just a word of caution: I hope this can be implemented without adding stat calls (or at least only when showing error messages or in other very slow code paths): in distributed filesystems repeated stat calls can literally kill performance.

[oschulz]

in distributed filesystems repeated stat calls can literally kill performance

Indeed! I'm not an expert on the internals of source file lookup, so I don't know how much this can be automatized. But for the use cases I have in mind, a "manual redirect" (via env-var, or a "path-remap TOML", etc.) would be perfectly fine too (telling Julia "/opt/julia-1.11 is now /cmvfs/unpacked.cern.ch/.../julia-1.11"). That would be stat-free.

Btw, another potential use case for re-located depots would be systems like NERSC that have virtual paths for "aggressively cached read-only" on cluster files systems.

[KristofferC]

I think it would be reasonable for Julia to persistently store only depot-relative (or potentially loaded-package-relative) paths and reconstruct them on demand.

We do this for the list of dependent files in a package image already:

https://github.com/JuliaLang/julia/blob/d6294ba973db1dea9dc932779008fd66d27c4bd2/base/loading.jl#L3353-L3361

https://github.com/JuliaLang/julia/blob/d6294ba973db1dea9dc932779008fd66d27c4bd2/src/staticdata_utils.c#L580-L587

This could likely be extended to the .file field of methods as well. Then this method:

https://github.com/JuliaLang/julia/blob/d6294ba973db1dea9dc932779008fd66d27c4bd2/base/methodshow.jl#L130-L148

just need to be updated to call

https://github.com/JuliaLang/julia/blob/d6294ba973db1dea9dc932779008fd66d27c4bd2/base/loading.jl#L3380-L3386

However, I am sure there are many direct .file field uses which would all instead have to go through the depot resolve step instead.

[vchuravy]

However, I am sure there are many direct .file field uses which would all instead have to go through the depot resolve step instead.

We should do that during deserialization of the package image.

[fatteneder]

We should do that during deserialization of the package image.

resolve_depot does at least one stats call. By calling it on every method added during loading we will run into the above stats issue. We were concerned about this already when merging the relocatable pkgimage PR (although I haven't heard anyone complain about it), so it might be time to think about a solution for that.