sc-controller icon indicating copy to clipboard operation
sc-controller copied to clipboard

Python2 -> Python3 (once more)

Open sunweaver opened this issue 4 years ago • 35 comments

Hi,

my name is Mike Gabriel, I am a Debian Developer and I was asked by the lead developer of Ubuntu Mate (Martin Wimpress) to bring this software to Debian and Ubuntu.

Unfortunately, I can't do this, as this project is still stuck with Python2. The Debian people are currently in the process of removing all Python2 packages from the distro with the goal to release Debian 11 in 2 years from now without Python2.

As of this, it is policy-wise forbidden since June 2019 to upload Python2 depending software to Debian. Please note that Python2 upstream is EOL in January 2020.

I noticed Python2 -> Python3 issues having been closed in the past with a "not gonna happen" kind of statement as the justification.

This issue report is to reconsider this tenor.

Thanks+Greets, Mike

sunweaver avatar Sep 23 '19 13:09 sunweaver

Hello,

there is only one Python2 -> Python3 issue, #367, and it should still be open. Nevertheless, your reading of "not gonna happen" is correct.

I'm currently in the process of rewriting "core" of SC-Controller into C, with code in c branch already being able to do big part of what Python code does. Thanks to that, I'm able to port SC-Controllers to systems that doesn't have issues like policy you've described.

Once "core" is finished, I'll have to decide if, and to what toolkit and language GUI part should be ported, but considering entire Py2/3 fiasco, Py3 is out of question. Since I'll have to bundle python interpreter on Windows anyway, bundling Py2 interpreter there and relying on availability of Py2 elsewhere looks like simplest solution.

Your distro deciding to drop Py2 support is indeed unfortunate, but it's not something I can influence in any way.

kozec avatar Oct 12 '19 18:10 kozec

Hi,

On Sa 12 Okt 2019 20:58:57 CEST, kozec wrote:

Hello,

there is only one Python2 -> Python3 issue, #367, and it should
still be open. Nevertheless, your reading of "not gonna happen" is
correct.

I'm currently in the process of rewriting "core" of SC-Controller
into C, with code in c
branch
already being
able to do big part of what Python code does. Thanks to that, I'm
able to port SC-Controllers to systems that doesn't have issues like
policy you've described.

Once "core" is finished, I'll have to decide if, and to what toolkit
and language GUI part should be ported, but considering entire Py2/3
fiasco, Py3 is out of question. Since I'll have to bundle python
interpreter on Windows anyway, bundling Py2 interpreter there and
relying on availability of Py2 elsewhere looks like simplest solution.

Thanks for the explanation. So, we will wait until the rewrite is available.

Your distro deciding to drop Py2 support is indeed unfortunate, but
it's not something I can influence in any way.

It is not only a distro thing. Py2 is EOL in Jan 2020 upstream. https://pythonclock.org/

Greets, Mike

DAS-NETZWERKTEAM c\o Technik- und Ökologiezentrum Eckernförde Mike Gabriel, Marienthaler str. 17, 24340 Eckernförde mobile: +49 (1520) 1976 148 landline: +49 (4351) 850 8940

GnuPG Fingerprint: 9BFB AEE8 6C0A A5FF BF22 0782 9AF4 6B30 2577 1B31 mail: [email protected], http://das-netzwerkteam.de

sunweaver avatar Oct 21 '19 13:10 sunweaver

A lot of distrubutions are aiming at removing python2 soon due to it going EOL next year. Still using python 2 will have the probability of bringing security concerns. I know for a fact that the diro that I'm using (Solus) are starting to deprecate stuff and when they are fully ready, everything that hasn't moved on will be dropped from the repo.

Jacalz avatar Oct 28 '19 08:10 Jacalz

Uhhh, Python2 is reaching EndOfLife, and will see no further development across the board.

https://www.python.org/doc/sunset-python-2/

So its not just a distro decision, its a Python.org decision. It will be deprecated / on life support everywhere.


What will happen if I do not upgrade by January 1st, 2020?

If people find catastrophic security problems in Python 2, or in software written in Python 2, then most volunteers will not help fix them. If you need help with Python 2 software, then many volunteers will not help you, and over time fewer and fewer volunteers will be able to help you. You will lose chances to use good tools because they will only run on Python 3, and you will slow down people who depend on you and work with you.

Some of these problems will start on January 1. Other problems will grow over time.


Basically no more security fixes, etc, by other package maintainers. Its dead.

DanielJoyce avatar Oct 30 '19 19:10 DanielJoyce

You can be stubborn all you want, but if you're wanting to ship python 2, you're pretty much gonna have to compile it yourself. As for fiasco, py2 being retired has been the plan since py3 was released. Its been what, a decade now?

DanielJoyce avatar Oct 30 '19 19:10 DanielJoyce

Hi, I'm maintaining the openSUSE experimental sc-controller package at games:tools. As @sunweaver and @Jacalz said it's a major problem relying on python 2.

Seil0 avatar Oct 31 '19 08:10 Seil0

If no one intends to port it to python3 a possible solution could be what the Oil shell does: taking a stripped down python2 interpreter with dependencies and bundle all of it in the project tree.

rnhmjoj avatar Oct 31 '19 15:10 rnhmjoj

taking a stripped down python2 interpreter with dependencies and bundle all of it in the project tree

Bundling End-of-Life software is hardly a good idea.

KAMiKAZOW avatar Nov 03 '19 10:11 KAMiKAZOW

If no one intends to port it to python3 a possible solution could be what the Oil shell does: taking a stripped down python2 interpreter with dependencies and bundle all of it in the project tree.

I think that's a problem for most linux distributions.

Seil0 avatar Nov 05 '19 07:11 Seil0

Currently porting to python3. The biggest issue is string encodings, but making steady progress.

Also JEDI caught a few bugs here and there.

Edit: Getting down to brass tacks I think.

Edit: Time to run and fix the tests, and break out a controller for testing. Found a missing dependency too for the builds.

Edit: Appindicator stuff seems kinda busted, especially on Linux/Gnome. Icon not showing, but dropdown works. Icon is blank right now. Still tracing that down. May just focus on tests for now, and leave that as todo. Build scripts aren't very bullet proof.

Also, lots of encoding changes, and windows will be fun because by default it tends to not use utf-8, so I suspect when I get something usable, that debugging that may be the most fun.

edit: Everyone has been sick at home. Checkout my username for fork branch

DanielJoyce avatar Dec 04 '19 17:12 DanielJoyce

@DanielJoyce Thanks a bunch, I'll try it out! Maybe consider submitting a PR.

MadeOfMagicAndWires avatar Jan 31 '20 12:01 MadeOfMagicAndWires

I did not know about this problem with python2 and ended up opening a bug report at Ubuntu's launchpad: https://bugs.launchpad.net/ubuntu/+source/python-pylibacl/+bug/1876350

@DanielJoyce have you given up on your attempt? Your fork seems the same for at least 6 months ago, yours don't even have the 0.4.9.10 release from Dec 25, 2019 from @kozec

Please don't let this software die. I know the steam controller has been stopped but I still find it the best controller ever. I have 132 different configurations for it currently, and I don't think Steam's driver is as good as kozec's.

Patola avatar May 04 '20 09:05 Patola

@Patola His port is kind of hidden. The upgrade-to-python3 branch is listed in stale branches on the fork but I ended up finding it via the Network Graph. The current port did not work for me anyway. The GUI comes up but it cannot detect my controller. Also, that fork has not been updated since 2019/12/07.

https://github.com/DanielJoyce/sc-controller/tree/upgrade-to-python3

I ended up spending some time last night to attempt to get it working on my own; I found the upgrade-to-python3 branch after the fact. I have a mostly working build of version 0.4.7 running under Python 3.7.7 on Fedora 31. Device detection works, the mapper works, and most portions of the OSD work; getting the keyboard working properly was a pain. Hotplugging devices does not seem to work.

I need to properly fork this repo and make a new branch with these changes.

Ryochan7 avatar May 09 '20 20:05 Ryochan7

HI,

On Sa 09 Mai 2020 22:25:45 CEST, Travis Nickles wrote:

@Patola His port is kind of hidden. The upgrade-to-python3 branch is
listed in stale branches on the fork but I ended up finding it via
the Network Graph. The current port did not work for me anyway. The
GUI comes up but it cannot detect my controller. Also, that fork has
not been updated since 2019/12/07.

https://github.com/DanielJoyce/sc-controller/tree/upgrade-to-python3

I ended up spending some time last night to attempt to get it
working on my own; I found the upgrade-to-python3 branch after the
fact. I have a mostly working build of version 0.4.7 running under
Python 3.7.7 on Fedora 31. Device detection works, the mapper works,
and most portions of the OSD work; getting the keyboard working
properly was a pain. Hotplugging devices does not seem to work.

I need to properly fork this repo and make a new branch with these changes.

This is great news. Thanks for working on this.

@flexiondotorg: highlighting you here as I know that you are probably
also interested in the Python3 port of sc-controller.

Mike

DAS-NETZWERKTEAM c\o Technik- und Ökologiezentrum Eckernförde Mike Gabriel, Marienthaler Str. 17, 24340 Eckernförde mobile: +49 (1520) 1976 148 landline: +49 (4351) 850 8940

GnuPG Fingerprint: 9BFB AEE8 6C0A A5FF BF22 0782 9AF4 6B30 2577 1B31 mail: [email protected], http://das-netzwerkteam.de

sunweaver avatar May 09 '20 21:05 sunweaver

Got the current master branch code ported to Python 3. It can still use some work but it is usable in its current state.

https://github.com/Ryochan7/sc-controller/tree/python3

Update 2020/05/10: A few more bugs fixes have been made. Profile exporting works. Also, device hotplugging works now.

Ryochan7 avatar May 10 '20 00:05 Ryochan7

Wow... Thank you, guys. Will test it ASAP.

Patola avatar May 11 '20 03:05 Patola

@Ryochan7 Thank's for your work. I was able to test it and found a few issues with the daemon, but i was able to fix them. Would you like me to open a pr at your repository?

Seil0 avatar May 13 '20 12:05 Seil0

@Seil0 Sure thing

I got the KeyGrabber class fixed. Ended up migrating some of my old custom changes; those will only be available as a Gist later. The experience is better and I hope to finally beat some games in my Linux backlog.

Updated Gist.

sccchanges_20200523.diff: https://gist.github.com/Ryochan7/316a644935f33e4f51ee19058c79777c

Ryochan7 avatar May 23 '20 14:05 Ryochan7

This frankly just makes me sad. This was by far the very best controller configurator/driver I have found, not only for my Steam controller, but for controllers in general. Has one of the Python3 forks been settled upon as the best option going forward? Which one should I switch to if I wish to continue using SC-Controller, but cannot continue using Python2 much longer? Ryochan7's fork perhaps?

Personally, I still don't understand the stubbornness surrounding the Python2 to 3 transition. I've been using Python3 for ages now, and literally the only issues I've had with Python3 thus far has come down entirely to a rare few developers bein' stubborn like this. In every other respect, Python3 has been a fantastic experience for me compared to Python2.

blooalien avatar Jun 04 '20 20:06 blooalien

The DanielJoyce fork didn't work for me when I tested it. At this time, it looks like my fork is the main Python 3 compatible fork of sc-controller. I plan on continuing working on the Python 3 fork for a while and keep the code in sync with this upstream master branch. It will be a okay stopgap measure until the c branch version is more full featured.

sc-controller has been the only piece of software that has made the Steam Controller a viable Xbox 360 controller alternative for me; even Steam Input didn't do what I wanted. I have wanted to get more use out of my Steam Controller again but Fedora switched to using Python 3 for the default Python installation. I would have hoped somebody else in the community would have taken a more active part beforehand rather than expecting Kozec to drop his current work though.

I would consider contributing a bit to the development of the c branch version but I cannot get the current c branch version to run; I can get version 0.4.9.10 built and running though. I will have to keep trying.

Ryochan7 avatar Jun 05 '20 21:06 Ryochan7

Hi Travis,

On Fr 05 Jun 2020 23:16:10 CEST, Travis Nickles wrote:

The DanielJoyce fork didn't work for me when I tested it. At this
time, it looks like my fork is the main Python 3 compatible fork of
sc-controller. I plan on continuing working on the Python 3 fork for
a while and keep the code in sync with this upstream master branch.
It will be a okay stopgap measure until the c branch version is more
full featured.

sc-controller has been the only piece of software that has made the
Steam Controller a viable Xbox 360 controller alternative for me;
even Steam Input didn't do what I wanted. I have wanted to get more
use out of my Steam Controller again but Fedora switched to using
Python 3 for the default Python installation. I would have hoped
somebody else in the community would have taken a more active part
beforehand rather than expecting Kozec to drop his current work
though.

I would consider contributing a bit to the development of the c
branch version but I cannot get the current c branch version to run;
I can get version 0.4.9.10 built and running though. I will have to
keep trying.

Would you be ok with being an upstream for Debian regarding the
sc-controller fork?

Mike (aka sunweaver at debian.org)

DAS-NETZWERKTEAM c\o Technik- und Ökologiezentrum Eckernförde Mike Gabriel, Marienthaler Str. 17, 24340 Eckernförde mobile: +49 (1520) 1976 148 landline: +49 (4351) 850 8940

GnuPG Fingerprint: 9BFB AEE8 6C0A A5FF BF22 0782 9AF4 6B30 2577 1B31 mail: [email protected], http://das-netzwerkteam.de

sunweaver avatar Jun 05 '20 21:06 sunweaver

Would you be ok with being an upstream for Debian regarding the sc-controller fork?

As long as Kozec has no problem with it then it is fine. I would probably hold off until at least the pull request from strycore is merged. For the most part, only Steam Controller support works. Support for other controller types still needs to be fixed.

Also, after a lot of struggling, I finally got the current c branch program running. Some of my problems were attributed to updates to MSYS2 that the release script does not take into account. Some libraries needed to be added to the release folder in order for libgtk-3-0.dll to load.

https://github.com/tschoonj/GTK-for-Windows-Runtime-Environment-Installer/issues/24

One other significant problem is due to PyGObject no longer supporting Python 2. As long as sc-controller keeps using Python 2, PyGObject will have to be frozen at version 3.34.0 and it will have to be installed with pip.

Ryochan7 avatar Jun 08 '20 18:06 Ryochan7

Got the changes from strycore, and others, merged into the python3 branch

Ryochan7 avatar Jun 09 '20 03:06 Ryochan7

Did it work for you? I tried to clone your repository, checked out the python3 tree, then built the appimage. But then:

[9:54] [9281] [patola@risadinha sc-controller]% ./SC_Controller-x86_64.AppImage&
[1] 42210
[9:54] [9282] [patola@risadinha sc-controller]% Traceback (most recent call last):
  File "/tmp/.mount_SC_Cono43XTU/usr/bin/scc", line 6, in <module>
    from scc.scripts import main
  File "/tmp/.mount_SC_Cono43XTU/usr/lib/python2.7/site-packages/scc/scripts.py", line 125
    print("Unknown profile:", argv[1], file=sys.stderr)
                                           ^
SyntaxError: invalid syntax

(zenity:42219): GdkPixbuf-WARNING **: 09:54:58.968: Cannot open pixbuf loader module file '/usr/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache': No such file or directory

This likely means that your installation is broken.
Try running the command
  gdk-pixbuf-query-loaders > /usr/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache
to make things work again for the time being.

(zenity:42219): Gtk-WARNING **: 09:54:59.275: Failed to set text 'Traceback (most recent call last):
  File "/tmp/.mount_SC_Cono43XTU/usr/bin/scc", line 6, in <module>
    from scc.scripts import main
  File "/tmp/.mount_SC_Cono43XTU/usr/lib/python2.7/site-packages/scc/scripts.py", line 125
    print("Unknown profile:", argv[1], file=sys.stderr)
                                           ^
SyntaxError: invalid syntax' from markup due to error parsing markup: Error on line 7 char 37: Element “markup” was closed, but the currently open element is “module”

(zenity:42219): GdkPixbuf-WARNING **: 09:54:59.304: Cannot open pixbuf loader module file '/usr/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache': No such file or directory

This likely means that your installation is broken.
Try running the command
  gdk-pixbuf-query-loaders > /usr/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache
to make things work again for the time being.

(zenity:42219): Gtk-WARNING **: 09:54:59.304: Could not load a pixbuf from icon theme.
This may indicate that pixbuf loaders or the mime database could not be found.

(zenity:42219): GdkPixbuf-WARNING **: 09:54:59.305: Cannot open pixbuf loader module file '/usr/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache': No such file or directory

This likely means that your installation is broken.
Try running the command
  gdk-pixbuf-query-loaders > /usr/lib/gdk-pixbuf-2.0/2.10.0/loaders.cache
to make things work again for the time being.
**
Gtk:ERROR:../../../../gtk/gtkiconhelper.c:494:ensure_surface_for_gicon: assertion failed (error == NULL): Failed to load /org/gtk/libgtk/icons/48x48/status/image-missing.png: Unrecognized image file format (gdk-pixbuf-error-quark, 3)
Bail out! Gtk:ERROR:../../../../gtk/gtkiconhelper.c:494:ensure_surface_for_gicon: assertion failed (error == NULL): Failed to load /org/gtk/libgtk/icons/48x48/status/image-missing.png: Unrecognized image file format (gdk-pixbuf-error-quark, 3)
/tmp/.mount_SC_Cono43XTU/AppRun: line 23: 42219 Aborted                 "$@"

[1]  + exit 1     ./SC_Controller-x86_64.AppImage

Patola avatar Jun 09 '20 07:06 Patola

I created a symbolic link from /usr/lib/x86_64-linux-gnu/gdk-pixbuf-2.0 to /usr/lib/gdk-pixbuf-2.0 but the application still fails:

[10:03] [9302] [patola@risadinha sc-controller]% ./SC_Controller-x86_64.AppImage 
Traceback (most recent call last):
  File "/tmp/.mount_SC_ConbPZI7m/usr/bin/scc", line 6, in <module>
    from scc.scripts import main
  File "/tmp/.mount_SC_ConbPZI7m/usr/lib/python2.7/site-packages/scc/scripts.py", line 125
    print("Unknown profile:", argv[1], file=sys.stderr)
                                           ^
SyntaxError: invalid syntax

(zenity:42741): Gtk-WARNING **: 10:03:11.241: Failed to set text 'Traceback (most recent call last):
  File "/tmp/.mount_SC_ConbPZI7m/usr/bin/scc", line 6, in <module>
    from scc.scripts import main
  File "/tmp/.mount_SC_ConbPZI7m/usr/lib/python2.7/site-packages/scc/scripts.py", line 125
    print("Unknown profile:", argv[1], file=sys.stderr)
                                           ^
SyntaxError: invalid syntax' from markup due to error parsing markup: Error on line 7 char 37: Element “markup” was closed, but the currently open element is “module”

Patola avatar Jun 09 '20 08:06 Patola

@Patola are you running it with python2?

rnhmjoj avatar Jun 09 '20 14:06 rnhmjoj

Did it work for you? I tried to clone your repository, checked out the python3 tree, then built the appimage. But then:

@Patola I would assume that it is trying to use Python 2. I guess the next task would be to try to update the appimage build script to take Python 3 into account.

Ryochan7 avatar Jun 09 '20 17:06 Ryochan7

It took some work but I got the AppImage version to work properly again. Quite a few changes needed to be made to get the AppImage version working. The AppImage version has been tested on Fedora 32, Manjaro 20.0.3, and Xubuntu 19.10. Hopefully it works fine on other distros as well.

SC_Controller-x86_64_20200613.AppImage https://drive.google.com/file/d/1l4I_Rnt092e2Zfqw_edLFcZtm54g9jpo/view?usp=sharing

Ryochan7 avatar Jun 13 '20 18:06 Ryochan7

Just tested this appimage, it's not working for me on Ubuntu 20.04. When it starts, if I have the steam controller on, it turns it off. If I turn it on again, it is not recognized. And when I exit the program, it seems to reset my USB subsystem, making my USB mouse become unresponsive.

In contrast, the regular sc-controller python2 appimage works correctly and does not do any of this.

Patola avatar Jun 13 '20 21:06 Patola

I tested the AppImage on a vanilla install of Ubuntu 20.04 and it worked for me. Although, support for vanilla Ubuntu 20.04 and later doesn't matter that much to me. As long as snapd remains proprietary, I will not recommend Ubuntu to anyone anymore. At least derived distros like Linux Mint are making snap an optional component.

Ryochan7 avatar Jun 14 '20 19:06 Ryochan7

support for vanilla Ubuntu 20.04 and later doesn't matter that much to me. As long as snapd remains proprietary, I will not recommend Ubuntu to anyone anymore.

Snapd is not proprietary. It's here on GitHub, licensed under the GPL v3: https://github.com/snapcore/snapd

xordspar0 avatar Jun 15 '20 13:06 xordspar0

I was mistaken about which portion is proprietary. The backend server is proprietary but the client side program is not.

Ryochan7 avatar Jun 16 '20 07:06 Ryochan7

Updated AppImage with the GtkHeaderBar tweak to remove the duplicate window icon when run on Plasma. Also, merged the DS4 v2 udev rule from upstream.

SC_Controller-x86_64_20200707.AppImage: https://drive.google.com/file/d/16QHJ4d5eaCYCYPdmFQ96DWJLh81L7gX8/view?usp=sharing

Ryochan7 avatar Jul 08 '20 13:07 Ryochan7

Looks like Ryochan7 took over.

Anyways, yes, the main dev branch for my work was https://github.com/DanielJoyce/sc-controller/tree/upgrade-to-python3

This had a lot of upgrades and updates pulled in. But its kinda stale by now.

DanielJoyce avatar Aug 17 '20 16:08 DanielJoyce

So... Whose build should we use?

Patola avatar Aug 18 '20 08:08 Patola