make a project environment and manage LOAD_PATH for JuliaPackage

[lexming]

trafficstars

This PR changes a bit how installations of JuliaPackages are handled to get the following improvements:

Fixes https://github.com/easybuilders/easybuild-easyconfigs/issues/19281 Alternative fix for https://github.com/easybuilders/easybuild-easyconfigs/pull/19650

• Properly handle JuliaPackages that have other JuliaPackages in their dependencies. Currently those dependencies are not seen by Julia.Pkg which results in packages being re-installed even though they can be found in the deps. This is fixed by adding all Julia packages found across dependencies to the Julia environment of the installation.
• Don't make assumptions about DEPOT_PATH and grab it from the Julia environment. This avoids needing JULIA_DEPOT_PATH in the shell environment.
• Respect user environment: user depot and its shared environment for this version of Julia are kept as top paths of DEPOT_PATH and LOAD_PATH respectively. This ensures that the user can keep using its own environment after loading JuliaPackage modules, installing additional software on its personal depot while still using packages provided by the module:
  • Don't reset DEPOT_PATH or LOAD_PATH on module load unless they are not set at all
  • Append path to installation to DEPOT_PATH and path to installation environment to LOAD_PATH

[lexming]

@boegelbot please test @ generoso EB_ARGS="IJulia-1.24.0-Julia-1.8.5.eb MultilevelEstimators-0.1.0-GCC-11.2.0-Julia-1.7.2.eb Circuitscape-5.12.3-Julia-1.9.2.eb Circuitscape-5.12.3-Julia-1.7.2.eb RCall-0.13.17-foss-2022a-R-4.2.1-Julia-1.9.2.eb"

[boegelbot]

@lexming: Request for testing this PR well received on login1

PR test command 'EB_PR=3239 EB_ARGS="IJulia-1.24.0-Julia-1.8.5.eb MultilevelEstimators-0.1.0-GCC-11.2.0-Julia-1.7.2.eb Circuitscape-5.12.3-Julia-1.9.2.eb Circuitscape-5.12.3-Julia-1.7.2.eb RCall-0.13.17-foss-2022a-R-4.2.1-Julia-1.9.2.eb" EB_CONTAINER= EB_REPO=easybuild-easyblocks /opt/software/slurm/bin/sbatch --job-name test_PR_3239 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

• exit code: 0
• output:

Submitted batch job 13053

Test results coming soon (I hope)...

- notification for comment with ID 1985233802 processed

Message to humans: this is just bookkeeping information for me, it is of no use to you (unless you think I have a bug, which I don't).

[boegelbot]

Test report by @boegelbot

Overview of tested easyconfigs (in order)

• SUCCESS IJulia-1.24.0-Julia-1.8.5.eb
• SUCCESS MultilevelEstimators-0.1.0-GCC-11.2.0-Julia-1.7.2.eb
• SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
• SUCCESS Circuitscape-5.12.3-Julia-1.7.2.eb
• FAIL (build issue) RCall-0.13.17-foss-2022a-R-4.2.1-Julia-1.9.2.eb (partial log available at https://gist.github.com/boegelbot/d36b92e87518153280ec344c81135c49)

Build succeeded for 4 out of 5 (5 easyconfigs in total) cns1 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8 See https://gist.github.com/boegelbot/119aea737428f95a9a8d32c93c5d15a1 for a full test report.

[lexming]

@boegelbot please test @ generoso EB_ARGS="IJulia-1.23.3-Julia-1.6.7.eb IJulia-1.24.0-Julia-1.8.5.eb MultilevelEstimators-0.1.0-GCC-11.2.0-Julia-1.7.2.eb Circuitscape-5.12.3-Julia-1.9.2.eb Circuitscape-5.12.3-Julia-1.7.2.eb"

[boegelbot]

@lexming: Request for testing this PR well received on login1

PR test command 'EB_PR=3239 EB_ARGS="IJulia-1.23.3-Julia-1.6.7.eb IJulia-1.24.0-Julia-1.8.5.eb MultilevelEstimators-0.1.0-GCC-11.2.0-Julia-1.7.2.eb Circuitscape-5.12.3-Julia-1.9.2.eb Circuitscape-5.12.3-Julia-1.7.2.eb" EB_CONTAINER= EB_REPO=easybuild-easyblocks /opt/software/slurm/bin/sbatch --job-name test_PR_3239 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

• exit code: 0
• output:

Submitted batch job 13096

Test results coming soon (I hope)...

- notification for comment with ID 1994749688 processed

Message to humans: this is just bookkeeping information for me, it is of no use to you (unless you think I have a bug, which I don't).

[boegelbot]

Test report by @boegelbot

Overview of tested easyconfigs (in order)

• SUCCESS IJulia-1.23.3-Julia-1.6.7.eb
• SUCCESS IJulia-1.24.0-Julia-1.8.5.eb
• SUCCESS MultilevelEstimators-0.1.0-GCC-11.2.0-Julia-1.7.2.eb
• SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
• SUCCESS Circuitscape-5.12.3-Julia-1.7.2.eb

Build succeeded for 5 out of 5 (5 easyconfigs in total) cns3 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8 See https://gist.github.com/boegelbot/7e0fbae412c57a421a187b7aa73523f1 for a full test report.

[branfosj]

Test report by @branfosj

Overview of tested easyconfigs (in order)

• SUCCESS IJulia-1.23.3-Julia-1.6.7.eb
• SUCCESS IJulia-1.24.0-Julia-1.8.5.eb
• SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
• SUCCESS Circuitscape-5.12.3-Julia-1.7.2.eb

Build succeeded for 4 out of 4 (4 easyconfigs in total) bear-pg0105u03a - Linux RHEL 8.6, x86_64, Intel(R) Xeon(R) Platinum 8360Y CPU @ 2.40GHz (icelake), Python 3.6.8 See https://gist.github.com/branfosj/59be7c3f84039927157e18aa85d69f75 for a full test report.

[branfosj]

Going in, thanks @lexming!

[branfosj]

Going in, thanks @lexming!