Thoughts about CY2019 and Qt.py

[fredrikaverpil]

I just added testing with CY2018 update 1 as a PR, although I don't think we need to merge it in its current state as it doesn't provide any extra value as of now. I merely completed it as I wanted to learn how to solve some issues entailed with setting that new container up.

However, when completing that PR, some ideas sprung up, which I'd like to raise.

sip

For the upcoming CY2019, we want to up the PyQt5 version substantially, and make it on par with PySide2. This would mean using Qt 5.12, PySide2-5.12, PyQt5-5.12. There is one major issue with this, and that is that there's not a sip version compatible with such a recent PyQt5 version and PyQt4. To my knowledge only one sip-version can be installed in an environment and both PyQt bindings will use that sip.

And the way we are currently running tests is that in each Python installation (2.7, 3.4, 3.5, 3.6), we are also installing sip, PyQt4, PyQt5, PySide (shiboken) and PySide2 (shiboken2). This has worked so far, as we have been using a sip/PyQt4/PyQt5 compatible combo cocktail. In retrospect, I should've seen this coming as there's no 🎲 that PyQt4 will be updated to support the most recent sip.

I will get back to the topic of sip further down below...

There are now official wheels, so why build for vfxplatform?

In the very early days of Qt.py, we just installed Python wheels and didn't muck around with building stuff from scratch. This, however, resulted in loss of control when e.g. Ubuntu PPAs were no longer maintained. CI/testing started breaking and in order to fix this, we decided to build software from individual git commits and specific software versions. By storing such builds as docker containers in dockerhub, we could be pretty sure that these configurations would last even if software grew old and their respective maintainers would abandon old versions.

So, beginning with CY2017, we've been building Qt from scratch so that we can control exactly which versions of bindings we are testing against. At this point we also decided why not build according to the vfxplatform spec. But in my mind, following that spec was secondary to building everything from source for increased control.

Building software from source seems like an obsolete scenario to me now (and an increasingly painstakingly daunting task), as PySide2 is just about to be stable and have a maintained version string (PySide2.__version__). Sure, it may not be built with the exact spec stipulated by the vfxplatform, and it won't contain any Autodesk-specific modifications. But do we really have to care about this? We aren't really testing Qt.py for Maya in particular and I think it's time to let loose and go DCC agnostic and simply make sure that Qt.py play nice with the vanilla bindings. We can always fix stuff which may arise due to the fact that a DCC app is misbehaving and add appropriate tests with mock data for such cases.

We are starting to tread out of the muddy waters of being the pioneers developing with PySide2, which feels great. I would like to embrace this fully and when moving forward, I wish we could just pip install wheels rather than build stuff from scratch like we've done with past containers.

My proposal moving forward

• Drop testing for PyQt4 and PySide in future testing (but keep the ones we have). We already have containers with these bindings and there are not much to gain from maintaining newer versions of these as development for PySide has stopped and drastically slowed down for PyQt4. It's just a pain to build these moving forward.
• Keep using the current nosetests suite (it works, why change it?)
• Start pip installing wheels (PyQt5 and PySide2) rather than building from source and rely on past docker containers to complete tests for the older bindings. Wheels would be stored somewhere so that they can be re-downloaded later.
• Possibly keep the current docker container creation workflow which would allow for building software if this need would arise again in the future.

Issues resulting from the above proposed changes:

• Today, we generate a dictionary of common binding members by running build_membership.sh. The way this works is that a members .json is dumped for each binding. A new .json is generated, containing only the members found in all previously generated json files. In the past, we've used the same docker container to produce these. With the changes above, we'll have to make a specific container dump the "legacy" bindings .json files and then use the latest container to dump the PySide2/PyQt5 .json files.
• entrypoint.sh can be modified to that it will not run tests for the "legacy" bindings unless a container running equipped with said bindings is executing.

---

@mottosso @MHendricks @Ahuge @dgovil and others - what do you think? PS. Anyone of you going to SIGGRAPH this year, would ❤️ to catch a 🍺 or 🍻 !

[dgovil]

So my thoughts:

• We've run into the same issue with the sip versions and it is very annoying that PyQt decided to keep sip as a shared name between incompatible bindings..

• Yes I think it is a good time to drop testing for PyQt4/Pyside in the future. In fact I'd advocate for maybe creating a version of Qt.py that focuses solely on Qt5 bindings. I feel the current Qt.py version has been sufficient for studios to port from Qt4 to Qt5 and can stay as the tagged release for Qt4 compatibility, but Qt5 is adding many things that makes it hard to support across Qt4 and Qt5 bindings. I do see a need for something to abstract between PyQt5 and PySide2 though till one becomes more ubiquitous. Maybe a version 2 of Qt.py would make sense following a deprecation period of PySide/PyQt4 testing in Qt.py 1.5+

• I like nosetests personally and see no benefit in switching from it.

• I think using the wheels makes sense rather than building from source. That would greatly simplify things. I also agree that there is no need to build against Maya's Qt because the tests don't get run in Maya and the changes aren't that significant.

---

Speaking of Autodesk's Qt version, I heard at one point there were plans to merge their changes into mainline Qt. Any idea if that's still on the table?

Also regarding deprecating Qt4, I realize that's not what you were initially suggesting, but I feel like I'm hitting more requests from TDs where they want functionality that is Qt5 specific. I think there may be more value gained in becoming a Qt5 only abstraction in the future, which is why I posit it here.

r.e. SIGGRAPH, I'm not 100% certain I will be there this year (sad because it's just down the street from me) but if I am, I'm definitely up for hanging out if I am in town!

[Ahuge]

I agree with you and @dgovil , I am not sure about the proposal for making it Qt5 aligned, but I see your point in a need for a Qt5 abstraction in my day to day work too.

• Nosetests works for me, I have no strong feelings about the specific test software
• I agree with using the wheels, and we should keep the containers around so that they can be specifically run if needed as well.

---

re: SIGGRAPH, I am pretty sure I will be attending, but just like @dgovil, I am just down the road and would love to catch up for a beer even if I cannot attend.

[dekekincaid]

@dgovil Most of Autodesk's changes are getting merged but usually they are only in newer versions then what vfxplatform has decided to settle on. Also most of the time those merges take 6-12 months to do and there is always some new patch from Autodesk which will only make it in some future release of Qt.

[dgovil]

@dekekincaid Would you know which version? The VFX Platform is going for 5.12 next which will be the latest for this year

[mottosso]

only one sip-version can be installed

We aren't limited by this technically I think; can't we can two sips in two different folders, and expose the Qt 4 version when testing PyQt4 and PySide, and the Qt 5 version otherwise?

E.g.

$ export PYTHONPATH=/path/to/oldsip
$ run_qt4_tests.py
$ export PYTHONPATH=/path/to/newsip
$ run_qt5_tests.py

Or, dynamically add the path during tests. Or, import the module by filename rather than import sip. Technically, we could do it.

Maybe a version 2 of Qt.py would make sense following a deprecation period of PySide/PyQt4 testing in Qt.py 1.5+

I've got a Qt5.py in play for a while that I've been thinking of making public, the only hesitation is how simple it is; the differences are far fewer than those between Qt 4 and 5 and putting one together ad-hoc and on-site is not an unreasonable thing to do.

However, there is value in community and sharing knowledge.

I wouldn't change Qt.py from being a Qt 4-5 wrapper however, I can see that becoming too confusing for users whereby they are using Qt.py, but for some users it doesn't work with Qt 4, and some it does. "Check your version" could become the standard reply to the most asked question online; "why isn't it working".

[Ahuge]

@mottosso folder scoping is totally possible and something that we've done at studios in the past for setting specific versions of software. We've done it for sip before so it can work

The import by filename may not work, I'm not sure..I think the new sip has other runtime libs that can't be switched during the same interpreter session. Although that may just be something weird with our setup.

Also I agree that Qt5.py should be separate than Qt.py because of the reasons you stated.

[MHendricks]

I haven't done any Qt5 only code yet, but it makes sense to separate it from Qt.py for when you want to take advantage of the new features.

I think using PYTHONPATH to control what sip is being loaded should work. I haven't really tried changing the version of sip on the same system, but I've definitely used different msvc compiled versions of sip on the same system by manipulating sys.path and the PATH environment variable on windows.

On linux we had to use the LD_LIBRARY_PATH environment variable instead of PATH to control how some dynamic libraries were loaded, though this was for Vray in nuke and not Qt.

---

re: SIGGRAPH, sorry, I won't be in town

[dekekincaid]

Renaming the project Qt5.py is a bit premature. Qt6 is not that far away in reality. Why not just major bump to 2.0 and have it as the Qt5 only version and still maintain the 1.x branch for the Qt4/Qt5. You loose a lot of history by going off and starting a separate project.

[fredrikaverpil]

on sip, folder scoping and PyQt4; PyQt4 cannot be pip-installed from official PyPi and must be built from source. This means we probably also will have to build Qt4 from source if we wish to keep installing PyQt4. Installing from PPA/Yum repo/etc means we might not be able to rebuild the container at a later date. All of the above is a pain point, and the only benefit I see from doing this is we can avoid changing how we do our membership/tests (which is not really in itself a necessity).

On a similar note, PySide(1) cannot be installed in anything above Python 3.4.

So, what we have today (the containers equipped with PyQt4 and PySide) works and attempting to recreate them moving forward is to me redundant as the bindings themselves doesn't change enough to warrant for the labor effort required.

So I wonder if we really will need to bother with folder scoping of sip, as we could just keep using our old containers and then new containers equipped with PySide2, shiboken2, PyQt5 and a PyQt5-compatible sip. Since there are official wheels for PySide2 and PyQt5 these container setups will be incredibly light-weight in comparison to what we have today.

---

@Ahuge @dgovil cool, I'll drop you an email privately sometime in Aug to try and meet up in Vancouver then!

[fredrikaverpil]

on Qt5-only Qt.py: As a professional developer, I would prefer we continue in this repository and start using branches and maintain a 1.x and 2.x version of Qt.py, like suggested by @dekekincaid.

But I also know that there are a lot of junior TDs or even senior artists who doesn't fully get how this works... and they simply browse into the github repo and click the download button. Then they just take the Qt.py file and drop that somewhere without having knowledge on that the repo wouldn't necessary hold the major version they needed.

Like @dekekincaid mentions, what happens when Qt6 is released? Perhaps it would be wise to indeed open a new repository, but in there always keep that Qt binding wrapper up to date to the very latest Qt version only (meaning; no backwards compatibility stuff). Let's call that QtWrapper in this post for clarity. We could then maintain different branches, e.g. 5.0, 6.0 (matching the Qt major version) and so on and make it available as pip install QtWrapper==5 or pip install QtWrapper==6 But for this existing Qt.py repo, we will never change the core feature; backwards compatibility. Perhaps this repo will release a Qt.py 2.0 one day which deals with going from Qt5 to Qt6.

[fredrikaverpil]

I think it would be wise to keep an open dialog with The Qt Company on what their plans are for Qt6 and their "Qt for Python". Perhaps the Qt.py project(s) is now starting to claim the import Qt, import Qt5, import Qt6 (and so on) namespaces.

ping @crmaurei !

[fredrikaverpil]

On Qt5.py...

Another approach is to keep adding new QtN.py files into this very existing "Qt.py" repo, which are locked to a Qt major version. So for example Qt5.py for PySide2/PyQt5 and down the road a Qt6.py for PySide3/PyQt6 so to speak. And then keep Qt.py the street smart one, handling backwards compatibility issues.

This would make it easier to run the same tests on the entire suite of Qt[N].py files and keep all issues within the same repo, offer the same extensibility options (QT_PREFERRED_BINDING, QtSiteConfig) etc. But this approach could perhaps also accumulate a great deal of cruft over the years...

But this cannot be combined with separate semver of Qt.py, for example if we decide it should from one point in time start supporting Qt5/Qt6 compatibility, as we'd have to sync the QtN.py files between branches. So... maybe all of this was not such a good idea. What I mean is you can't easily pip install a certain version of Qt.py and then pip install another certain version of QtN.py if they are part of the same release.

EDIT: I see @mottosso started another discussion here: https://github.com/mottosso/Qt.py/issues/294

[dgovil]

@fredrikaverpil I don't believe your ping crmaurei worked? But I agree with your point on not having a Qt5/Qt6 if there's a risk that the Qt foundation would want to use those names. I imagine at some point they'd want to deprecate the PySide name and go for a simpler, from Qt5 import QtWidgets and it would be good to not have a name clash there.

Personally I would be for a major version bump to 2.x.x instead.

[fredrikaverpil]

Ah... A typo. Ping @cmaureir - please see https://github.com/mottosso/Qt.py/issues/293#issuecomment-402369551

[cmaureir]

Hello, thanks for inviting me to your discussion.

AFAIK the renaming got postponed because many people complained, and also we didn't want the user base of PySide to think this was something else, and not an updated version of it.

From the last discussions, there is two options for a future renaming. PySideX or QtX (where X is the version), sadly there is no final decision on this matter (schedule or final name), but I will be more than happy to tell you as soon as we started to discuss the topic again.

Keep in mind that the official release of Qt for Python is theoretically scheduled for Qt 5.12, so it's probable that we have a decision before that.

Of course, you are more than welcome to ask about the status of this on the IRC channel.

[fredrikaverpil]

Thanks @cmaureir for this. So it wouldn't be far fetched to say you are planning on making a future PySide actually be importable using import QtX (where X is e.g. the major Qt version).

@mottosso I think we are shooting everyone's feet by naming the "next-gen" wrapper Qt5.py as it will then be used as import Qt5 and will be really confusing once PySide changes its name.

[cmaureir]

I will try to start this discussion internally to see if we will be able to make a decision soon, so you can be aware of it.

[boredstiff]

I work at a studio that uses Qt.py and we incorporated it back in January/February throughout hundreds of thousands of lines of code.

While I'm not completely opposed to the needs of dropping the support for Qt4, and I can see why you would want to do it, I heavily agree with the motion to major version bump it as opposed to renaming it. Doing so would allow us to choose when to drop our support for Qt4 which isn't slated for about half of our code base.

Currently we have a monorepo that has library source code in one area that touches inside of DCCs and is the Qt5/PySide2 compliant area that we've fixed, and we have another section that is called 'apps' that is, and will for the foreseeable future, continue to be Qt4-compliant and has not been run heavily or tested on Qt5 yet (and we cannot get the resources to transition this). However - this app area has been already refactored and it is using Qt.py just like the library area, but we don't have PySide2 in the environment that these apps use, so it's still using PyQt4.

We have already spent a ton of time refactoring our monorepo and the hundreds of thousands of lines of code to import Qt.py - If you major version bump this and don't make it a separate package/module that we have to import from, you allow us to keep our code base in it's current form and drop Qt4 when we want and not have to do another major refactor.

[mottosso]

If you major version bump this and don't make it a separate package/module that we have to import from, you allow us to keep our code base in it's current form and drop Qt4 when we want and not have to do another major refactor.

That is a good point. If there's one thing to look out for with that it would be that although you will at some point be able to drop Qt 4, you would not be able to use any features unique to Qt 5 until that has happened.

Conversely, with a separate Qt5.py, you could employ those features projects right away and in parallel to your existing projects.

But there will come a time when Qt 6 is out and we'll need a 5-6 wrapper. When that happens, it's no longer clear what Qt.py means, and I'm not entirely convinced a relatively hidden aspect like version is a strong enough deliminator for someone reading code and know without a doubt that it's using Qt 4, 5 or 6.

Perhaps Qt.py should be Qt4to5.py, followed by Qt5to6.py? Not as catchy, but more explicit. Then at some point, if we're lucky, we'd get Q T 9 to 5, what a way to make a livin'

[boredstiff]

Yeah, I can see your point there.

I don't know if there's a case for us with using a Qt5.py because we have 0 plans to integrate PyQt5/future PyQts while the Qt company is planning to maintain PySide2/Qt for Python (please don't let them call it that :smile: ) We would instead just import from PySide2 if we need specific features (which I currently do right now).

We would definitely use a Qt5to6 wrapper should we ever get to that point in the future, though. Although I hope that's not for a few years. But as we all know, the VFX industry will be a decade behind whatever is current so we've probably got time. :)

[dgovil]

Depending how this conversation goes, I think it may be worth posting this on the vfx-reference platform or other similar mailing lists, to get inputs from other studios. I can see advantages and disadvantages to both.

If it helps for future abstractions, at Imageworks, I've set it up so we wrap Qt.py inside a package that dynamically passes through. So with our set up we could switch to a different Qt.py version at runtime easily or switch to a differently named module like Qt5.py at anytime without changing our imports in our dependent packages that have been ported to use our package.

[dekekincaid]

Just an additional thumbs up again to doing a major version bump for us. We just spent the last 9 months transitioning hundreds of packages and thousands of man hours to Qt.py. 40+ td's/engineers did all this work migrating to this package. A lot of this took so long because we had lots of pyqt4 api1 code. So it wasn't exactly the Qt.py part that was the expensive side. The thing is having to now migrate it to Qt5.py is a bit disheartening to us as we just finished a majority of this transition a little over a month ago.

I have not consulted with the other supervisors but I suspect if this project goes this route we would strongly consider forking Qt.py and backport Qt5.py changes to our own Qt.py. This is not a threat just a reality of the cost of doing business. Small changes such as this are very expensive to larger organizations. If you did this over a year ago before VFXPlatform 2017 when everyone had to transition it would have been a smaller deal but now everyone is in full swing and probably has mostly already moved to qt5 packages and @fredrikaverpil mentioned, I strongly thing this is shooting yourselves in the foot.

[mottosso]

Thanks for sharing, @dekekincaid.

For clarity, in an ideal world what would such a fork do differently? For example, would it drop support for Qt 4, and add features unique to Qt 5, such as the Multimedia and QML modules? What is your stance on the issue raised above, about not being able to leverage Qt 5 until all use of Qt 4 has been removed (due to not being able to import both 1.0 and 2.0 into the same namespace)? Is it an option to add features unique to Qt 5 into Qt.py, whilst still also supporting Qt 4?

[Ahuge]

@mottosso

I think @dekekincaid was talking primarily about the namespace change. What may seem like a small change on the Qt.py side, adding another repo with a slightly different name (Qt5), would actually result in a large amount of work for us to change all of our code to import Qt5 instead of Qt.

Going the major version route would also allow us to name the github releases for clarity but code-side still "import Qt" regardless of the version we use.

[dekekincaid]

@mottosso we have a package management system, so we can simply pin different versions for different shows/roles(similar to Rez). If there were a 2.0 we would just pin that version packages which use qt5. As @Ahuge mentions if you change the namespace we now have to upgrade 150 different packages before moving from Qt to Qt5 namespace and probably add hundreds of ugly try excepts to handle backward compatibility. It would be faster for us to simply patch the single location of Qt5 to be Qt rather then updating the hundreds of packages.

Also I don’t see the benefit to importing both Qt and the proposed Qt5 at the same time. What is the benefit of having both?

[mottosso]

What is the benefit of having both?

Sorry for not being more clear, the benefit is being able to use modules that support Qt 4 alongside modules that utilise features of Qt 5, such as QtMultimedia and QtQml.

For example, if application A, B and C runs in Maya 2015-2018 and you wanted to develop application D to utilise QtMultimedia - let's say it's a new playblasting and review tool. Unlike A, B and C, D will only be used on a new show where all DCCs have access to Qt 5; but you'd still like to use A, B and C.

• Scenario 1: Qt.py continues to support Qt 4-5, and application D uses Qt5.py instead of Qt.py
• Scenario 2: Qt.py 2.x drops support for Qt 4, and limits A, B and C to DCCs with Qt 5

If there were a 2.0 we would just pin that version packages which use qt5

The above assumes you import Qt without tricks.

import Qt

Whereby modules go by the "there can be only one" rule, as the next import Qt from another module, such as another application, would use the same version as the one imported elsewhere.

If I understand what you mean, you are suggesting something along these lines?

• Scenario 3: Qt.py 2.x drops support for Qt 4, versions 1.x and 2.x are accessible via a namespace

By which I mean..

from v1.Qt import QtWidgets

class ApplicationA(QtWidgets.QDialog):
   ...

from v2.Qt import QtMultimedia

class ApplicationD(QtMultimedia.Player):
  ...

Everlasting Life

On a separate but related note, would it be interesting to have Qt.py support Qt Widgets indefinitely? That is, when Qt 6 comes out, Qt.py soldiers on with support for Qt 4-6; assuming Qt hasn't dropped widgets. I figure, if Qt 6 does drop widgets, there would be no escaping another major refactor anyway, and perhaps this way the work put into developing with widgets today would last as long possible.

[MHendricks]

What about using both Qt.py and QtX.py modules?

from Qt import QtCore, QtWidgets
from Qt5 import QtMultimedia, QtQml

Qt5.py would just cover the Qt 5 only parts of Qt, and Qt is still used to access the majority of the Qt namespace.

We could use major versions of Qt.py to control what versions of Qt they support. v1 supports compatibility between Qt 4 and Qt 5. v2 adds support for Qt 6 but drops support for Qt 4.

This gives you a easy way to add support for features that are not shared between multiple versions, but still keep import refactoring to a minimum. Changing Qt.py to v2(dropping support for Qt4) may not be needed depending on how much Qt changes when it goes to Qt 6. Ideally we could put off Qt.py v2 for a long time.

Depending on how much things change in Qt 6, it's also possible that Qt5.py could be used to wrap Qt 6 in the same way as Qt.py. It would always follow the namespace of Qt 5 but account for Qt 6 namespace changes. Qt6.py would use the Qt 6 namespace, but include any new features that Qt5.py supports but Qt.py does not support.

from Qt import QtCore, QtWidgets
# QtWidgets contains new classes that are not in Qt.py
from Qt6.QtWidgets import QtNewClass

I can see QtX.py working two ways:

1. It's fully featured, you can just use QtX.py for all Qt access.
2. It only wraps the classes that are not supported in previous versions of Qt.

I'm leaning towards the second one as it prevent's making your code require a complete refactor when upgrading Qt versions (changing all imports from Qt5 to Qt6).

I think that QtX.py would respond to the same environment variable config, so setting QT_PREFERRED_BINDING = 'PySide2;PyQt5' will affect it the same way. It would probably raise a exception if you were using a unsupported binding QT_PREFERRED_BINDING = 'PySide;PyQt4'.

[fredrikaverpil]

Having previously discussed the mentioned solution above (by @MHendricks) with @mottosso, I think this is the way to go.

In my mind, there's really just one thing to this approach which needs to be sorted out, and that is how to keep track of the versions/releases in Github:

• Would we ever want to be able to release a Qt.py 1.x update if we are currently on Qt.py 2.x?
• Do we desire to be able to pip install Qt.py and pip install QtX.py separately from each other?

To enable both these features, the way I see it is that we could maintain different branches; 4to5 and 5to6 (for Qt.py), 5 and 6 (for the QtX.py ones) etc. Then each branch has its own CHANGELOG.md. There would not be a master but the most actively developed one will be the "default" branch.

We would then be able to create releases for each branch:

[fredrikaverpil]

Hm. I'm not entirely sure we can release a 1.x version on PyPi if we are on 2.x. Does anyone know?

EDIT: I'm lazy and on vacation, here's an SO question which hopefully can answer this.

[mottosso]

Hold on, I'm not sure this addresses the concern brought up by @dekekincaid and @alexwidener.

Their concern, and correct me if I'm wrong, was to keep their existing codebase unspoiled when introducing features unique to Qt 5, such as QtMultimedia. If we limit support of Qt 5 to what is currently available in Qt.py, and instead implement those features in a new project such as Qt5.py, then although their codebase would remain secure, but they would be unable to leverage Qt 5 unless they added Qt5.py on-top.

Was that right, @dekekincaid and @alexwidener?