rez-bind python erroneously set some PYTHONPATH for python/lib and python/extra

[gst]

Hi,

I discovered, already some days/week ago, that rez-bind python actually creates a package.py which contains instructions to add to PYTHONPATH the paths to 2 of the "main" libs directories of the python which was bound to. (it's the directories where the os and setuptools modules are available under the python which is bound to). It (rez-bind) also has copied everything under these 2 paths into the package installation directory of that python.

We so end with :

../packages/python/x.y.z/platform-linux/arch-x86_64/bin/python -> link to the real python binary which was bound to.

But:

../packages/python/x.y.z/platform-linux/arch-x86_64/python/lib
../packages/python/x.y.z/platform-linux/arch-x86_64/python/extra

are now also in PYTHONPATH.

BUT, now anything which use a python under the hood/internally will have to use that PYTHONPATH containing directories to the selected (by rez-env) python own system libs/modules.

We can see that effect like this :

$ python -m site
sys.path = [
    '/home/gregory',
    '/home/gregory/rez/master-2.9.1/packages/python/2.6.6/platform-linux/arch-x86_64/os-CentOS-6.7/python/lib',
    '/home/gregory/rez/master-2.9.1/packages/python/2.6.6/platform-linux/arch-x86_64/os-CentOS-6.7/python/extra',
    '/usr/lib64/python26.zip',
    '/usr/lib64/python2.6',
    '/usr/lib64/python2.6/plat-linux2',
    '/usr/lib64/python2.6/lib-tk',
    '/usr/lib64/python2.6/lib-old',
    '/usr/lib64/python2.6/lib-dynload',
    '/home/gregory/.local/lib/python2.6/site-packages',
    '/usr/lib64/python2.6/site-packages',
    '/usr/lib64/python2.6/site-packages/PIL',
    '/usr/lib64/python2.6/site-packages/gst-0.10',
    '/usr/lib64/python2.6/site-packages/gtk-2.0',
    '/usr/lib64/python2.6/site-packages/webkit-1.0',
    '/usr/lib64/python2.6/site-packages/wx-2.8-gtk2-unicode',
    '/usr/lib/python2.6/site-packages',
]
USER_BASE: '/home/gregory/.local' (exists)
USER_SITE: '/home/gregory/.local/lib/python2.6/site-packages' (exists)
ENABLE_USER_SITE: True

As you can see, given they are in PYTHONPATH, they take precedence over my .local (USER_SITE) directory.. which is a first not good.

But, if I would have rez-env with something using a python under hood, like nuke for instance, then Houston we have a (big) problem :

because nuke uses its own python, not necessarily compatible with the one/thoses I've bound to.

That is : nuke starts, it starts its own embedded python, which early import many important modules, like openssl, but then boum because the openssl for one python version isn't compatible with another. and given PYTHONPATH the python under nuke happens so to import the important modules from the python that was resolved in my rez-env call, not from his own python.

It seems to me you're trying, somehow, to do a kind of virtualenv manually (but only a kind of) but the way it is actually triggers the problem(s) I mention / try to explain here. But PYTHONPATH is not done for that. well I mean putting a python system library/directory in PYTHONPATH is calling for problems.. as soon as you have to play with different python versions.

Actually, only doing the link to the python which is bound-to is enough to make it all available.

waiting answer / feedback or question ..

I can propose the PR (which is simply to not do the copy of these dirs and not add them to PYTHONPATH in the package.py which is generated) if you agree with this ..

[bpabel]

It should only add those paths to the PYTHONPATH if you do

rez-env python

Or if one of you package files lists that python as a dependency. The nuke package shouldn't be loading the rez python package.

If your packages need to list a specific version of python as a dependency (for the purpose of variants), you can list them implicitly, where it won't actually load the package.

In your package.py file, you would do

requires = ['~python-2.7+']

That way variants of compiled python libraries like ssl should choose the correct variant, without actually loading the local python package.

[gst]

without going into too deep details,

rez-env python (with whatever version specification),

doesn't need/require the overhead that is actually done inside rez/bind/python.py (to do the same effect than if it had not that overhead). on the other side keeping it creates problems.

:s

[gst]

otherwise the current code of the binding, actually uses the system python (well the one available on PATH) to copy its files from (from the 2 directories identified by os and setuptools modules). Not the one which is actually bound to. We end up with the python binary of one python version, but (some of) the system modules of another python version.. (I'm using export PATH=path_to_my_rez_installation_rez_subdirectory(:$PATH) : # export PATH=/home/gregory/rez/master-2.9.1/bin/rez:$PATH as given/told by rez when you python install.py it (with /home/gregory/rez/master-{version} arg in this case). )

I already fixed that in my rez clone some days/week ago up to when I realized it is not necessary at all, more over it implies the hard problems that I describe even with that side problem fixed. and also mentioning that it actually so duplicates the system modules where os.py is located (python2.7 directory most of the time) + the site-packages directory (where setuptools most of the time is there).

Hopefully there is no need of this copy/duplicates. I'm running with that for few days and that solves completely the problem(s) I mention.

really PYTHONPATH is not designed to put the "system" modules directory of a particular python version. A particular python version knows where its system modules are (unless you've manually copied parts - but not all - of a python installation, which you can imagine is not good). it should only contains third-party packages/modules.

[nerdvegas]

Yo, so a couple of things.

First, I have no problem removing those system paths if that doesn't cause further problems, I'll take a look.

Some background: Packages created with rez-bind aren't really intended for production use, and as such they haven't gotten a lot of love. Python especially - the rez-bind of python is really just a helper to get up and running quickly. You could do a proper installation of python from source and wrap that into a rez package instead: https://github.com/nerdvegas/rez/tree/master/repository/python/2.7.4.

You've touched on a bigger topic when talking about Nuke already containing python. We know there are lots of cases like this, and the challenge is how to deal with vendor-supplied packages in a consistent way. One idea is to introduce the concept of a package 'providing' another package - the nuke package could 'provide' python-2.N, which would then cause rez to drop the standard python package and let nuke deal with it. It's possible to deal with this situation already though, although not quite as elegantly. Consider a python package.py like so:

# python 
def post_commands():
    if defined(env._PYTHON_PROVIDED):
        return
    ...

With this approach, you could have nuke require ~python-2.N.N (as @bpabel suggested), but also have it set _PYTHON_PROVIDED, and do whatever other (potentially build-related) things are necessary to expose its own copy of python for other packages to use. Here python only implements post_commands(), to ensure that other packages have a chance to set _PYTHON_PROVIDED first.

[instinct-vfx]

This is an interesting problem that gets even more interesting on windows as often ;)

Mostly due to the fact that I have to provide custom-built binaries for binary python extensions in nuke (and other DCCs) and even multiple ones for different versions in some cases due to changing toolchains and python versions. That means that for example there needs to be a nuke specific version of lxml. This creates all sorts of funny problems. For example what if someone wanted an environment to run both nuke and some command line python tools that are launched from nuke. eek. This is something we are discussing currently and some of the discussion here seems helpful. Thanks!

[gst]

@nerdvegas

Python especially - the rez-bind of python is really just a helper to get up and running quickly.

ok got it. and effectively it's only a simple wrapper to provide access, via rez, to some already available/installed python.

I have no problem removing those system paths if that doesn't cause further problems

I think it doesn't cause further problems. but I'm sure it solves many of them (and which are quite rude to me).

I've read further your comment,

maybe I've to explain a bit how I see my current use case(s) :

I would like to be able to have a rez-env with many tools requested, with some of them being purely binary/compiled (but which can embed their own python), while others which could be python scripts (or any other script language). But I would like to be able to run some of the scripts with a particular python(or any other language) version ; say tool X can run with python2.7.3 or 2.7.4 while tool Y wants to run with python2.6.9 (ok or say 2.7.9, we should be able to forget about 2.6 very soon) ; while others with the greatest python version available (maybe with a major specifier at least: python2.*) (available via a rez package ofcourse).

If we can make available a specific python version with python-major.minor.patch alias (or link being on PATH to the real binary) then we are able to use different python versions in the same env. and "python" would be the alias (or link) to the actual greatest version available (and resolved).

but I know this isn't something currently possible (to have different versions of the same package in the same env).. I was just wondering..

Going a bit further : actually each tool could depend on different versions of the same third-party libraries and I would like to be able to handle that.

I see it like that : for each tool : have a wrapper script (being prepended on PATH and named the same than the actual tool, for consistency, simplicity) inside the env that actually setups all what it needs.

hmm, not sure about all that..

but anyway : we deviate slightly from the purpose of this issue ..

[nerdvegas]

Hey Gregory,

This is basically what 'suites' are for. Rez specifically will resolve to an environment containing one version of each package, that's called a "context". A "suite" is exactly what you describe here:

"I see it like that : for each tool : have a wrapper script (being prepended on PATH and named the same than the actual tool, for consistency, simplicity) inside the env that actually setups all what it needs."

With a suite, you can store multiple contexts in it; and wrapper scripts for the tools in those contexts are created, and put into a single 'bin' dir. When you run the wrapper, it boots into the associated context, and starts the tool.

Suites rely on shebanged scripts though, so they only work on linux/osx, I don't recall if you're one of those Windows people or not...!

Take a look into the forum, this has come up before and I've written up examples of how to create a suite with the 'rez-suite' cli tool. I need to add this to the docs.

Hth A

On Fri, Mar 17, 2017 at 2:01 AM, Grégory Starck [email protected] wrote:

@nerdvegas https://github.com/nerdvegas

Python especially - the rez-bind of python is really just a helper to get up and running quickly.

ok got it. and effectively it's only a simple wrapper to provide access, via rez, to some already available/installed python.

I have no problem removing those system paths if that doesn't cause further problems

I think it doesn't cause further problems. but I'm sure it solves many of them (and which are quite rude to me).

I've read further your comment,

maybe I've to explain a bit how I see my current use case(s) :

I would like to be able to have a rez-env with many tools requested, with some of them being purely binary/compiled (but which can embed their own python), while others which could be python scripts (or any other script language). But I would like to be able to run some of the scripts with a particular python(or any other language) version ; say tool X can run with python2.7.3 or 2.7.4 while tool Y wants to run with python2.6.9 (ok or say 2.7.9, we should be able to forget about 2.6 very soon) ; while others with the greatest python version available (maybe with a major specifier at least: python2.*) (available via a rez package ofcourse).

If we can make available a specific python version with python-major.minor.patch alias (or link being on PATH to the real binary) then we are able to use different python versions in the same env. and "python" would be the alias (or link) to the actual greatest version available (and resolved).

but I know this isn't something currently possible (to have different versions of the same package in the same env).. I was just wondering..

Going a bit further : actually each tool could depend on different versions of the same third-party libraries and I would like to be able to handle that.

I see it like that : for each tool : have a wrapper script (being prepended on PATH and named the same than the actual tool, for consistency, simplicity) inside the env that actually setups all what it needs.

hmm, not sure about all that..

but anyway : we deviate slightly from the purpose of this issue ..

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/399#issuecomment-287085044, or mute the thread https://github.com/notifications/unsubscribe-auth/ABjqSlfrgE51lYOZitQFZXz_0YwoLcoiks5rmU7OgaJpZM4MdKID .

[gst]

hey @nerdvegas

ok I'll have to look and try the suites then. thx.

Suites rely on shebanged scripts though, so they only work on linux/osx, I don't recall if you're one of those Windows people or not...!

we have all: linux + osx + windows ;)

[herronelou]

Hello @nerdvegas. I'm doing a brand new setup of rez, and got into a similar situation as @gst above (and his suggestion to not include the pythonpath for the python package does seem to solve the issue). As I keep setting up packages, I'm running into a few situations where having a "provides" keywork would really come in handy.

For example I'm setting up Qt.py right now. It needs either PySide2, PyQt5, PySide or PyQt4 to run. I could make these variants, but Nuke provides its own PySide (1 or 2 depending on version) so I wouldn't really need to add the dependency. I could make weak requires for all 4 packages, but then I can't just 'rez-env qtpy'. I was also looking at the option to do a late requires() with checks to if in_context() and "nuke" in request:, but I would need to do conditions for all dccs, and I don't want my qt package to be aware of my dccs. I can't figure out what the best approach is, but it sounds to me like the 4 variants would be the correct approach, with nuke indicating that it 'provides' PySide2.

I'm sure I'm not the first person hitting this situation but the documentation isn't super clear to me as what the best approach is. @mottosso has quite a few examples that are complementing the documentation nicely with his own bleeding-rez fork, and I wish I could give a try to rez-scoopz but the machines don't have internet access.

[herronelou]

Found out the above is #1100

[gst]

@nerdvegas

Is Rez still adding/inserting into PYTHONPATH some/any python system libs directory(ies) ?

if yes I'll leave this open. If not I'll close ;)