julia icon indicating copy to clipboard operation
julia copied to clipboard

WIP: refactor the code loading of packages

Open KristofferC opened this issue 3 years ago • 3 comments

The current code for code loading is quite stateless (although there is currently a cache for parsed TOML files). Every time information about a package is looked up, the code loading system starts from scratch by searching through the load path, reading through manifests looking for the piece of information it needs. Touching disk a lot was a performance problem that is now mostly worked around in https://github.com/JuliaLang/julia/pull/40890 but the solution in there is quite ugly and ad hoc.

This PR explores another approach where the full information of an environment is parsed and stored in an object (which then makes lookups trivial). This is a bigger initial cost but the idea is that environments change rarely enough that this can be cached over multiple calls into code loading (as well as getting passed to precompile workers) which will save work in the long run.

This PR so far implements the new lookup strategy but it still needs the caching part where the environment stack is persisted over several calls to require as well as getting passed to precompile workers. This will also need a PkgEval run since there might be quite a few packages using the internals of code loading (maybe Revise for example).

For ease of reading, the new stuff is in a separate file called codeloading2.jl but this will be folded into loading.jl when this is done.

KristofferC avatar Sep 09 '22 13:09 KristofferC

This will replace the https://github.com/JuliaLang/julia/compare/jn/loading-plus branch (https://github.com/JuliaLang/julia/commit/8ca3f3b2fa636d3ec506f0a1f454a385721b794f), right?

vtjnash avatar Sep 09 '22 13:09 vtjnash

Yes, that's the idea.

KristofferC avatar Sep 09 '22 13:09 KristofferC

Awesome. I will delete that then now

vtjnash avatar Sep 09 '22 14:09 vtjnash