Make python executable a variable in pypresso script

[SamTov]

The pypresso script has the absolute path written several times. If someone installs espresso inside of a conda environment and then changes it or wants to use pypresso with different environments they would either have to re-build it in those environments or change all of the calls to python in the pypresso script. If however python was a path variable, this could be changed once at the start of the script and then be used for all environments.

Alternative option, check if python is in the current path and if not, default back to a full path so users need not have any information about environments unless python is not in their path by default.

[jngrad]

I'm not sure if I understood your request correctly, as we store the absolute path to many dependencies.

How would you suggest amending src/python/pypresso.cmakein to support different environments?

Please keep in mind that we have the following constraints:

• the CMake input file must generate a valid launcher script for both ./pypresso and ./src/python/pypresso, which do not import the same ESPResSo shared objects
• both launchers must remain valid when calling cmake . to update any CMake option (variable PYTHON_DIR is overwritten twice with a different value, the pypresso and jupyter executable paths may be updated)
• there are other variables in the launchers that depend on the environment, in particular the paths to the pypresso and jupyter executable are hardcoded to avoid surprises when packaging on Fedora (what should happen on an OS with 2 versions of python available?)

[SamTov]

So I am not sure entirely about editing the cmakein file, but perhaps that is what is required. I just noted yesterday in the built pypresso executable at the start it is written:

if test -n "$PYTHONPATH"; then
  PYTHONPATH="/tikhome/stovey/work/Repositories/espresso/build/src/python:$PYTHONPATH"
else
  PYTHONPATH="/tikhome/stovey/work/Repositories/espresso/build/src/python"
fi
export PYTHONPATH

However, for the rest of the script there are a lot of absolute paths such as

case "$1" in
    --gdb)
        shift
        [ "/tikhome/stovey/miniconda3/envs/zincware/bin/python3" = "/tikhome/stovey/miniconda3/envs/zincware/bin/jupyter" ] && exec gdb -ex "set print thread-events off" -ex "set exec-wrapper sh -c 'exec \"/tikhome/stovey/miniconda3/envs/zincware/bin/jupyter\" \"\$@\"'" --args "/tikhome/stovey/miniconda3/envs/zincware/bin/python3" "$@"

laying around. I was helping someone use pypresso with some other larger python libraries and found that their install was done in a different conda environment and therefore, all of these paths were wrong. I thought theoretically, this could be done with a variable instead of hard-coding and therefore one could avoid users from having to reinstall espresso if they swap conda environments as it is a little more involved that installing regular packages.

If this is done by editing then the cmakein file then I will need to look a bit more to understand them better before commenting.

[jhossbach]

I've had the same issue, a change would be much appreciated

[bindgens1]

I had the same problems back in the day and resorted to rebuild espresso when changing the environment. This change would make handling espresso much easier.

Although I am not sure if the conda environments work (conda used to break everything and you end up with weird mixtures of system python and conda python). Regular python environments would already be beneficial.

[SamTov]

For most standard things I haven't had an issue with conda. From my experience when I see mixtures of system python and conda python it is usually in a standalone executable that can be pip installed such as jupyter. So maybe in the case of ipypresso this would need to be considered but it is more about the user ensuring that their ipython installation is also under the conda environment that they want to use.

[jngrad]

I thought about providing a new CMake flag -D PYTHON_ENV_INDEPENDENT=ON to replace hardcoded Jupyter/Python paths by strings jupyter and python3, in order to delegate executable selection to the shell. It's however unclear how to handle injection of the hardcoded path to the ESPResSo python files folder into the PYTHONPATH variable. The path is currently expanded at the beginning of the string such that it is the first folder to be traversed. If we expand it at the end to allow constructions like PYTHONPATH="/path/to/espresso" ./pypresso script.py, then we have the issue that any globally installed espresso package will be picked up before the local build. We could store the hardcoded path in a local variable that can be overridden by the user, but then there is not much point in using the pypresso wrapper script.

To all interested parties: how about using the following construction instead of the pypresso script?

PYTHONPATH="/work/name/espresso/build/src/python${PYTHONPATH:+:$PYTHONPATH}" python3 script.py
PYTHONPATH="/work/name/espresso/build/src/python${PYTHONPATH:+:$PYTHONPATH}" jupyter notebook

This is more or less how it's done on Fedora and OpenSUSE. You can write the python path logic as an exported variable in a module file to manage multiple installations via module load, or alternatively in a shell script that you would pass to source.

[jngrad]

Closing due to inactivity. The proposed PYTHONPATH solution is routinely used in Linux packages and Docker containers.