MycroftAI
Python 3.9+ issues
@j1nx reported that Exceptions were raised during Padatious training on python 3.9 similar to
Exception in thread Thread-8:
Traceback (most recent call last):
File "/usr/lib/python3.9/threading.py", line 950, in _bootstrap_inner
self.run()
File "/usr/lib/python3.9/threading.py", line 888, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.9/site-packages/padatious/training_manager.py", line 98, in train
pool = mp.Pool()
File "/usr/lib/python3.9/multiprocessing/context.py", line 119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
File "/usr/lib/python3.9/multiprocessing/pool.py", line 191, in __init__
self._setup_queues()
File "/usr/lib/python3.9/multiprocessing/pool.py", line 343, in _setup_queues
self._inqueue = self._ctx.SimpleQueue()
File "/usr/lib/python3.9/multiprocessing/context.py", line 113, in SimpleQueue
return SimpleQueue(ctx=self.get_context())
File "/usr/lib/python3.9/multiprocessing/queues.py", line 342, in __init__
self._rlock = ctx.Lock()
File "/usr/lib/python3.9/multiprocessing/context.py", line 67, in Lock
from .synchronize import Lock
ImportError: cannot import name 'Lock' from partially initialized module 'multiprocessing.synchronize' (most likely due to a circular import) (/usr/lib/python3.9/multip
rocessing/synchronize.py)
The issue seems to be related to a change in how modules/packages are imported from python 3.9 and onwards See bug report.
A quick and dirty workaround for testing is available in here. Basically it preloads the modules that may be incomplete when trying to create the Pool object.
It's a bit hard to tell exactly what is happening but one possible culprit is msm, which also use a multiprocessing Pool (threadpool but it's the same base class)...unless training is somehow launched in separate threads as well as processes internally in padatious but that sounds farfetched.
Not sure what the most correct action is a couple of things that can be tested:
- Lock around msm.apply calls and padatious training
- Move the execution of training to the same thread as the skill download (SkillManager thread)
To Reproduce Steps to reproduce the behavior:
- Install mycroft with a python3.9 environment
- Startup mycroft
- watch skills log
- Check for ImportError in multiprocessing
Environment (please complete the following information):
- Ubuntu 18.04
- Ubuntu 20.04
- Open Voice OS
I hope I soon have a separate OVOS image with Python 3.9 again for testing this. Will see if I can make a clean OVOS-Python3.9 build on another (temperarily) build machine. Report back as soon as possible.
Tried to install mycroft with Python 3.9:
When I start mycroft cli I got some errors:
Starting cli
Traceback (most recent call last):
File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/pi/mycroft-core/mycroft/client/text/__main__.py", line 19, in <module>
import curses
File "/usr/local/lib/python3.9/curses/__init__.py", line 13, in <module>
from _curses import *
ModuleNotFoundError: No module named '_curses'
With your fixed process_utils.py I got those errors:
Starting cli
Traceback (most recent call last):
File "/usr/local/lib/python3.9/runpy.py", line 188, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/usr/local/lib/python3.9/runpy.py", line 111, in _get_module_details
__import__(pkg_name)
File "/home/pi/mycroft-core/mycroft/__init__.py", line 17, in <module>
from mycroft.api import Api
File "/home/pi/mycroft-core/mycroft/api/__init__.py", line 22, in <module>
from mycroft.configuration import Configuration
File "/home/pi/mycroft-core/mycroft/configuration/__init__.py", line 15, in <module>
from .config import Configuration, LocalConf, RemoteConf
File "/home/pi/mycroft-core/mycroft/configuration/config.py", line 23, in <module>
from mycroft.util.json_helper import load_commented_json, merge_dict
File "/home/pi/mycroft-core/mycroft/util/__init__.py", line 32, in <module>
from .process_utils import (reset_sigint_handler, create_daemon,
File "/home/pi/mycroft-core/mycroft/util/process_utils.py", line 7
<!DOCTYPE html>
Is there a workaround or should I relink Python to 3.7 and reinstall mycroft?
It should work with python 3.9. I have it working, in fact, with python 3.9. There is the multiprocessing error, and I need to kill skills manually after stopping services, because they didn't stop properly, but is definitely working. Do you have installed curses-menu? it seems it complains about it
It should work with python 3.9. I have it working, in fact, with python 3.9. There is the multiprocessing error, and I need to kill skills manually after stopping services, because they didn't stop properly, but is definitely working.
Would like to +1 this behaviour. Indeed, with Python 3.9 stopping the mycroft-skills service takes a long time (till time out).
Has anyone tried the workaround branch I mentioned at the top? would be interesting to see if it makes things work better, or it's just for me(TM) :)
Not by myself as of yet. When the OVOS release is out of the door, I will get back to it.
Not me either, can I apply just this or need to do some git magic (if you can guide me to easily test it without the need to reinstall everything, would be great, I usually just clone, push and pull, so other git stuff are still a mystery for me)
Yeah that's what's the total change.
you could just copy-paste it. Or the following should work (written from memory without testing so it may be wrong)
git checkout -b py39-test
git pull https://github.com/forslund/mycroft-core.git bugfix/mp-workaround
Then to go back to dev you can just do
git checkout dev
git remote add forslund https://github.com/forslund/mycroft-core.git
git pull forslund
git checkout forslund/bugfix/mp-workaround
Found that easier for in case you test, give feedback, forslund changes some stuff again. Then all you need to do is
git pull
https://docs.python.org/release/3.9.2/library/multiprocessing.html#multiprocessing.pool.ThreadPool
Is this related?
Me neither, just saw the new 3.9.2 release and this change in documentation about old ways and to be changed to new ways. That together with multiprocessing and pool, got me wondering if those proposed changes might relate to how things are done within mycroft-core as well.
I can't tell for sure but I think it's the change in import machinery that's the root issue for Mycroft and we need to do some redesign to make sure the the inits of Pools aren't called from several threads. which will require some rethinking I believe.
Could also try to replace the multiprocessing module with the concurrent.futures module which offers similar functionality.
Edit: Or use the hack I linked above, I've been using it quite alot on my Python 3.9 desktop to get Mycroft running properly.
Will get to giving you hack a go soon. Within the next few days, buildroot 2021.02 LTS will be released. Then I will bump and first compile run will be without the python downgrade. If all good I can leave it like that using 3.9.2 This will make sure all latest CVE's are pulled in.
I did a couple tests today with concurrent.futures creating the following branches:
mycroft-skills-manager feature/replace-mp padatious feature/replace-mp
msm is based upon the 0.9.0 version so it will require the PRs from @PureTryOut
Right, I am onto testing this Python 3.9 stuff again. OVOS is fully bumped at the moment, so would like to give all changes a go.
Can you quickly explain to me about the best way forward? Which patches to which repo's are needed at this point?
Sorry for the late reply.
Alt 1: the two linked PR's in padatious and msm. MSM has some XDG-related updates so I'm not sure it will work without the matching XDG updates in core (but maybe you have those in OVOS?) padatious has an issue if the training hangs it won't abort correctly but should work mostly. (was planning on looking at that this week but some other work got in between)
Alt 2: Add this commit to mycroft-core. It should preload all the necessary parts.
Ok, alt2 seems simple. Anyhow, will see if I can give both a spin.
in your oppinion, which alt you think is the better way of dealing with it as the "real / beter" fix to be merged upstream?
I think alternative 1 is the more longterm solution. alt 2 is more of a workaround.