Allow custom shared projects for `julia_project`

[MilesCranmer]

trafficstars

This will fix #196. If you specify an @ in your julia_project string, like julia_project="@my_project", then this will be designated a shared project (just as if you had specified ]activate @my_project in the Julia REPL.

cc @mkitti

[mkitti]

Merge https://github.com/conda-forge/pysr-feedstock/pull/51 to unpin openlibm then the test here will pass

[MilesCranmer]

Ah, I think I see an issue (at least, for testing this). Right now the init_julia only sets the environment via an environment variable - rather than the activate() command. The only time activate() is used explicitly is in the install part: https://github.com/MilesCranmer/PySR/blob/b6ac303316bf72dbf49f29418c519e6dd2f120d9/pysr/julia_helpers.py#L86-L88

So I think what we should actually do is call activate() if Julia has already been initialized. (Since changing the environment variable won't change the env!)

[mkitti]

So I think what we should actually do is call activate() if Julia has already been initialized. (Since changing the environment variable won't change the env!)

Correct. The only reason I used the environment variable earlier was to select the environment where we expect PyCall to be, which may not be the default environment.

[MilesCranmer]

Okay so now the DEPOT_PATH correctly sets the project. However, for some reason, julia.install() switches the environment back to the normal DEPOT_PATH... Not sure why.

from pysr.julia_helpers import init_julia, install

Main = init_julia()
Main.eval('pushfirst!(DEPOT_PATH, "/tmp/1")')
Main = init_julia("@test2")

This correctly goes to the /tmp/1/environments. However! When you do:

install("@test2")

this sets to the normal DEPOT_PATH...

  Activating project at `~/.julia/environments/test2`

Is this something in PyJulia maybe?

[mkitti]

However! When you do:

install("@test2")

this sets to the normal DEPOT_PATH...

  Activating project at `~/.julia/environments/test2`

Is this something in PyJulia maybe?

In this case it seems you have not manipulated DEPOT_PATH, so I'm guessing that DEPOT_PATH is just the default value.

Note that the DEPOT_PATH global is specific to each Julia session. Manipulating it does not influence the corresponding environment variable JULIA_DEPOT_PATH. Thus if the environment variable JULIA_DEPOT_PATH is not set and you have not manipulated DEPOT_PATH, the depot will be at its default location.

[MilesCranmer]

Thanks - the weird thing is that those commands are from the same Python session. So the DEPOT_PATH should stick around through the install() call…

[mkitti]

Is this happening on all versions of Julia or just particular ones, perhaps less than Julia 1.7.0?

I'm thinking about the subprocess and _get_julia_env_dir stuff that we did. The Julia process run by subprocess would have no idea about changes you may have made within the current Julia process.

[MilesCranmer]

Okay I think that got it. It wasn't the subprocess - it was the JULIA_DEPOT_PATH that was missing! Good catch.

[MilesCranmer]

Cool:

  Activating project at `/var/folders/1h/xyppkvx52cl6w3_h8bw_gdqh0000gr/T/tmpmrkyi7cm/environments/pysr_test_env`

Not sure what the conda errors are though.

[MilesCranmer]

Ah, the conda errors were just because the environment.yml wasn't updated to the new Julia 1.8.0 requirement for openlibm - have since unpinned it and seems to be happy.

[mkitti]

This looks like an improvement to me. I still think we should figure out how to generalize this to non-shared environments at some point.