introduce easyconfig parameter for custom python paths

[smoors]

many easyconfigs currently have PYTHONPATH defined in modextrapaths, which escapes the --prefer-python-search-path=EBPYTHONPREFIXES setting.

updated proposal:

by introducing a new parameter, say python_paths, those paths can be added either to PYTHONPATH or to a new envvar, say EBPYTHONPATHS, depending on what --prefer-python-search-path is set.

the sitecustomize.py script will then add the EBPYTHONPATHS to sys.path (in addition to EBPYTHONPREFIXES/lib/pythonx.y/site-packages).

[Flamefire]

I very much welcome this as $PYTHONPATH breaks virtualenvs "override" behavior.

However they are not equivalent and we need to consider these cases:

1. EasyConfigs setting $PYTHONPATH to <prefix>/lib/pythonx.y/site-packages should just remove that as it is handled by the framework
2. Those setting $PYTHONPATH to <prefix>/any-sub-folder/lib/pythonx.y/site-packages might benefit from this new option as they can set that to python_paths = 'any-sub-folder
3. In all other cases our current EBPYTHONPREFIXES doesn't handle this

We could introduce a similar env variable EBPYTHONPATHS that takes full paths instead of prefixes. But that requires updating our script installed to the Python module which is currently only done during a rebuild.

[smoors]

ah you're right, i assumed that sitecustomize.py would add EBPYTHONPREFIXES to sys.path as is if the lib/pythonx.y/site-packages postfix doesn't exist, but that's clearly not the case.

introducing a EBPYTHONPATHS envvar that is equivalent to PYTHONPATH is probably the cleanest. we could already do this right now, and only start using the python_paths param in new toolchains.

i'll update the proposal above.

[Micket]

Related: https://github.com/easybuilders/easybuild-easyconfigs/issues/21456

In that issue I list the ones that make non-standard paths like this and couldn't make use of EBPYTHONPREFIXES. I would overall classify them as

1. poorly written installers that needless break things by adding extra crap into the path, like FSL prepending "fsl/"
2. just naively putting a python package into a randomly named directory.

I bet a lot of these could be fixed with maybe an extra postinstallcmds = ['mkdir blabla/site-packages', 'ln -s ../../foo blabla/site-packages/foo'] so that it could use EBPYTHONPREFIXES.