mitogen
mitogen copied to clipboard
Published
20 hours ago •
mitogen-hq
mitogen-hq
The Mitogen master process was unable to serve u'ansible_mitogen.target'. on ansible core 2.13
Running the latest master of mitogen, I'm getting the following error on all hosts I've tried against (Debian 10-11, CentOS 7-8, Ubuntu 20-22):
[WARNING]: Unhandled error in Python interpreter discovery for host: exceptions.ImportError: The Mitogen master process was
unable to serve u'ansible_mitogen.target'. It may be a native Python extension, or it may be missing entirely. Check the importer debug logs on the master
for more information. File "<stdin>", line 3698, in _dispatch_one File "<stdin>", line 3685, in _parse_request File "<stdin>", line 682, in
import_module File "<stdin>", line 1521, in load_module
Ansible version info:
ansible [core 2.13.2]
config file = /etc/ansible/ansible.cfg
configured module search path = ['/home/uumas/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3.10/site-packages/ansible
ansible collection location = /home/uumas/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/bin/ansible
python version = 3.10.5 (main, Jun 6 2022, 18:49:26) [GCC 12.1.0]
jinja version = 3.1.2
libyaml = True
Traceback:
Traceback (most recent call last):
File "/usr/lib/python3.10/site-packages/ansible/executor/task_executor.py", line 157, in run
res = self._execute()
File "/usr/lib/python3.10/site-packages/ansible/executor/task_executor.py", line 635, in _execute
result = self._handler.run(task_vars=vars_copy)
File "/home/uumas/.local/lib/python3.10/site-packages/ansible_mitogen/mixins.py", line 146, in run
return super(ActionModuleMixin, self).run(tmp, task_vars)
File "/usr/lib/python3.10/site-packages/ansible/plugins/action/normal.py", line 47, in run
result = merge_hash(result, self._execute_module(task_vars=task_vars, wrap_async=wrap_async))
File "/home/uumas/.local/lib/python3.10/site-packages/ansible_mitogen/mixins.py", line 376, in _execute_module
self._set_temp_file_args(module_args, wrap_async)
File "/home/uumas/.local/lib/python3.10/site-packages/ansible_mitogen/mixins.py", line 355, in _set_temp_file_args
self._connection.get_good_temp_dir()
File "/home/uumas/.local/lib/python3.10/site-packages/ansible_mitogen/connection.py", line 832, in get_good_temp_dir
self._connect()
File "/home/uumas/.local/lib/python3.10/site-packages/ansible_mitogen/connection.py", line 854, in _connect
self._connect_stack(stack)
File "/home/uumas/.local/lib/python3.10/site-packages/ansible_mitogen/connection.py", line 801, in _connect_stack
dct = mitogen.service.call(
File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/service.py", line 126, in call
return call_context.call_service(service_name, method_name, **kwargs)
File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/core.py", line 2314, in call_service
return recv.get().unpickle()
File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/core.py", line 977, in unpickle
raise obj
mitogen.core.CallError: builtins.ModuleNotFoundError: The Mitogen master process was unable to serve 'ansible_mitogen.target'. It may be a native Python extension, or it may be missing entirely. Check the importer debug logs on the master for more information.
File "<stdin>", line 3698, in _dispatch_one
File "<stdin>", line 3685, in _parse_request
File "<stdin>", line 682, in import_module
File "<stdin>", line 1521, in load_module
Some log (hostname replaced with ...):
[mux 22293] 12:58:33.935628 D mitogen.responder: sending ansible (0.53 KiB) to ssh....
[mux 22293] 12:58:33.937290 D mitogen.responder: ansible.module_utils is a package at /usr/lib/python3.10/site-packages/ansible/module_utils/__init__.py with submodules ['_text', 'ansible_release', 'api', 'basic', 'common', 'compat', 'connection', 'csharp', 'distro', 'errors', 'facts', 'json_utils', 'parsing', 'powershell', 'pycompat24', 'service', 'six', 'splitter', 'urls', 'yumdnf']
[mux 22293] 12:58:33.937960 D mitogen.responder: sending ansible.module_utils (0.48 KiB) to ssh....
[mux 22293] 12:58:33.999957 D mitogen: PkgutilMethod(): <_frozen_importlib_external.SourceFileLoader object at 0x7f67bd505db0>.get_file_name('ansible.module_utils.distro') failed: ImportError('loader for distro cannot handle ansible.module_utils.distro')
[mux 22293] 12:58:34.000093 D mitogen: SysModulesMethod(): <module 'distro' from '/usr/lib/python3.10/site-packages/distro/__init__.py'>.__name__ is 'distro', aborting
[mux 22293] 12:58:34.000192 D mitogen: ParentEnumerationMethod(): 'ansible.module_utils.distro' is PKG_DIRECTORY: '/usr/lib/python3.10/site-packages/ansible/module_utils/distro/__init__.py'
[mux 22293] 12:58:34.016629 D mitogen: While importing 'ansible.module_utils.basic'
Traceback (most recent call last):
File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 1134, in _send_module_and_related
tup = self._build_tuple(fullname)
File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 1087, in _build_tuple
for name in self._finder.find_related(fullname)
File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 959, in find_related
names = self.find_related_imports(name)
File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 933, in find_related_imports
set(
File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 937, in <genexpr>
and not is_stdlib_name(name)
File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 140, in is_stdlib_name
modpath = os.path.abspath(getattr(module, '__file__', ''))
File "/usr/lib/python3.10/posixpath.py", line 378, in abspath
path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType
[mux 22293] 12:58:34.017687 D mitogen.responder: sending ansible.module_utils.json_utils (2.68 KiB) to ssh....
[mux 22293] 12:58:34.018253 D mitogen.responder: sending ansible.release (0.99 KiB) to ssh....
[mux 22293] 12:58:34.018647 D mitogen.responder: ansible_mitogen is a package at /home/uumas/.local/lib/python3.10/site-packages/ansible_mitogen/__init__.py with submodules ['affinity', 'compat', 'connection', 'loaders', 'logging', 'mixins', 'module_finder', 'parsing', 'planner', 'plugins', 'process', 'runner', 'services', 'strategy', 'target', 'transport_config', 'utils']
[mux 22293] 12:58:34.018774 D mitogen.responder: sending ansible_mitogen (0.44 KiB) to ssh....
[mux 22293] 12:58:34.076591 D mitogen: While importing 'ansible_mitogen.runner'
Traceback (most recent call last):
File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 1134, in _send_module_and_related
tup = self._build_tuple(fullname)
File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 1087, in _build_tuple
for name in self._finder.find_related(fullname)
File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 959, in find_related
names = self.find_related_imports(name)
File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 933, in find_related_imports
set(
File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 937, in <genexpr>
and not is_stdlib_name(name)
File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 140, in is_stdlib_name
modpath = os.path.abspath(getattr(module, '__file__', ''))
File "/usr/lib/python3.10/posixpath.py", line 378, in abspath
path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType
[mux 22293] 12:58:34.094960 D mitogen: While importing 'ansible_mitogen.target'
Traceback (most recent call last):
File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 1134, in _send_module_and_related
tup = self._build_tuple(fullname)
File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 1087, in _build_tuple
for name in self._finder.find_related(fullname)
File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 959, in find_related
names = self.find_related_imports(name)
File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 933, in find_related_imports
set(
File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 937, in <genexpr>
and not is_stdlib_name(name)
File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 140, in is_stdlib_name
modpath = os.path.abspath(getattr(module, '__file__', ''))
File "/usr/lib/python3.10/posixpath.py", line 378, in abspath
path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType
[mux 22293] 12:58:34.158828 D mitogen.responder: sending mitogen.parent (18.87 KiB) to ssh....
[mux 22293] 12:58:34.159104 D mitogen.responder: sending mitogen.fork (2.63 KiB) to ssh....
[mux 22293] 12:58:34.191350 D mitogen.responder: sending mitogen.select (1.62 KiB) to ssh....
[mux 22293] 12:58:34.191604 D mitogen.responder: sending mitogen.service (8.90 KiB) to ssh....
well, not exactly a playbook but ansible localhost -m ping:
uumas@lemon ~> ansible localhost -m ping
[WARNING]: No inventory was parsed, only implicit localhost is available
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: File "<stdin>", line 1521, in load_module
localhost | FAILED! => {
"msg": "Unexpected failure during module execution.",
"stdout": ""
}
I'm afraid I'm unsable to reproduce
(v) ➜ mitogen git:(issue849-module_fqn) ✗ ANSIBLE_STRATEGY=mitogen_linear ansible localhost -m ping
[WARNING]: No inventory was parsed, only implicit localhost is available
localhost | SUCCESS => {
"changed": false,
"ping": "pong"
}
(v) ➜ mitogen git:(issue849-module_fqn) ✗ ansible --version
ansible [core 2.13.1]
config file = /Users/alex/src/mitogen/ansible.cfg
configured module search path = ['/Users/alex/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /opt/homebrew/Cellar/ansible/6.1.0/libexec/lib/python3.10/site-packages/ansible
ansible collection location = /Users/alex/.ansible/collections:/usr/share/ansible/collections
executable location = /opt/homebrew/bin/ansible
python version = 3.10.5 (main, Jun 23 2022, 17:14:57) [Clang 13.1.6 (clang-1316.0.21.2.5)]
jinja version = 3.1.2
libyaml = True
for me the ping example works as well, but I receive the traceback while facts gathering
TASK [Gathering Facts] ***************************************************************************************************************************
The full traceback is:
Traceback (most recent call last):
File "/usr/lib/python3.10/site-packages/ansible/executor/task_executor.py", line 157, in run
res = self._execute()
File "/usr/lib/python3.10/site-packages/ansible/executor/task_executor.py", line 635, in _execute
result = self._handler.run(task_vars=vars_copy)
File "/home/mitogen/ansible_mitogen/mixins.py", line 146, in run
return super(ActionModuleMixin, self).run(tmp, task_vars)
File "/usr/lib/python3.10/site-packages/ansible/plugins/action/gather_facts.py", line 100, in run
res = self._execute_module(module_name=fact_module, module_args=mod_args, task_vars=task_vars, wrap_async=False)
File "/home/mitogen/ansible_mitogen/mixins.py", line 376, in _execute_module
self._set_temp_file_args(module_args, wrap_async)
File "/home/mitogen/ansible_mitogen/mixins.py", line 355, in _set_temp_file_args
self._connection.get_good_temp_dir()
File "/home/mitogen/ansible_mitogen/connection.py", line 832, in get_good_temp_dir
self._connect()
File "/home/mitogen/ansible_mitogen/connection.py", line 854, in _connect
self._connect_stack(stack)
File "/home/mitogen/ansible_mitogen/connection.py", line 801, in _connect_stack
dct = mitogen.service.call(
File "/home/mitogen/mitogen/service.py", line 126, in call
return call_context.call_service(service_name, method_name, **kwargs)
File "/home/mitogen/mitogen/core.py", line 2314, in call_service
return recv.get().unpickle()
File "/home/mitogen/mitogen/core.py", line 977, in unpickle
raise obj
mitogen.core.CallError: builtins.ModuleNotFoundError: The Mitogen master process was unable to serve 'ansible_mitogen.target'. It may be a native Python extension, or it may be missing entirely. Check the importer debug logs on the master for more information.
File "<stdin>", line 3698, in _dispatch_one
File "<stdin>", line 3685, in _parse_request
File "<stdin>", line 682, in import_module
File "<stdin>", line 1521, in load_module
fatal: [monitor.systemlos.net]: FAILED! => {
"msg": "Unexpected failure during module execution.",
"stdout": ""
}
mitogen: current main ansible [core 2.13.2]
thanks for your work @moreati
Ansible 2.13.2 when doing "with_fileglob" on template task.
Using the master version of mitogen as of today.
- name: Copy template files
template: src={{item}} dest=/usr/local/folder/
with_fileglob:
- "../templates/*.conf"
register: var1
Exception ignored in: <function Connection.__del__ at 0x10a766310>
Traceback (most recent call last):
File "/Users/weber/git/mitogen/ansible_mitogen/connection.py", line 531, in __del__
self.close()
File "/Users/weber/git/mitogen/ansible_mitogen/connection.py", line 884, in close
self._put_connection()
File "/Users/weber/git/mitogen/ansible_mitogen/connection.py", line 865, in _put_connection
self.chain.reset()
File "/Users/weber/git/mitogen/mitogen/parent.py", line 1852, in reset
self.call_no_reply(mitogen.core.Dispatcher.forget_chain, saved)
File "/Users/weber/git/mitogen/mitogen/parent.py", line 1907, in call_no_reply
self.context.send(self.make_msg(fn, *args, **kwargs))
File "/Users/weber/git/mitogen/mitogen/core.py", line 2310, in send
self.router.route(msg)
File "/Users/weber/git/mitogen/mitogen/core.py", line 3383, in route
self.broker.defer(self._async_route, msg)
File "/Users/weber/git/mitogen/mitogen/core.py", line 2864, in defer
raise Error(self.broker_shutdown_msg)
mitogen.core.Error: An attempt was made to enqueue a message with a Broker that has already exitted. It is likely your program called Broker.shutdown() too early.
Have been facing this issue on a Nix environment with Python 3.11.8, Ansible 2.13.13 and Mitogen 0.3.4.
So I hijacked the is_stdlib_name() function a bit
diff --git a/mitogen/master.py b/mitogen/master.py
index 4fb535f0a..130d655af 100644
--- a/mitogen/master.py
+++ b/mitogen/master.py
@@ -137,7 +137,21 @@ def is_stdlib_name(modname):
return False
# six installs crap with no __file__
- modpath = os.path.abspath(getattr(module, '__file__', ''))
+ _g = getattr(module, '__file__', '')
+ if _g is None:
+ LOG.error('is_stdlib_name: modname: %r', modname)
+ LOG.error('is_stdlib_name: module: %r', module)
+ _attrs = dir(module)
+ LOG.error('is_stdlib_name: dir(module): %r', _attrs)
+ LOG.error('is_stdlib_name: __file__: %r', __file__)
+ LOG.error('is_stdlib_name(): getattr(module, \'__file__\', \'\'): %r', _g)
+ for _a in _attrs:
+ _g = getattr(module, _a, '')
+ LOG.error(' attr: %r: %r', _a, _g)
+
+ modpath = os.path.abspath(_g)
return is_stdlib_path(modpath)
ERROR! [mux 2268210] 22:35:45.446946 E mitogen: is_stdlib_name: modname: 'backports'
ERROR! [mux 2268210] 22:35:45.447049 E mitogen: is_stdlib_name: module: <module 'backports' (<_frozen_importlib_external.NamespaceLoader object at 0x7f5dd79bb290>)>
ERROR! [mux 2268210] 22:35:45.447200 E mitogen: is_stdlib_name: dir(module): ['__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'configparser']
ERROR! [mux 2268210] 22:35:45.447263 E mitogen: is_stdlib_name: __file__: '/nix/store/14yvxad8lv32kjb54hp1gsahyf89p6d6-python3.11-mitogen-0.3.4/lib/python3.11/site-packages/mitogen/master.py'
ERROR! [mux 2268210] 22:35:45.447281 E mitogen: is_stdlib_name(): getattr(module, '__file__', ''): None
ERROR! [mux 2268210] 22:35:45.447316 E mitogen: attr: '__doc__': None
ERROR! [mux 2268210] 22:35:45.447333 E mitogen: attr: '__file__': None
ERROR! [mux 2268210] 22:35:45.447349 E mitogen: attr: '__loader__': <_frozen_importlib_external.NamespaceLoader object at 0x7f5dd79bb290>
ERROR! [mux 2268210] 22:35:45.447367 E mitogen: attr: '__name__': 'backports'
ERROR! [mux 2268210] 22:35:45.447383 E mitogen: attr: '__package__': 'backports'
ERROR! [mux 2268210] 22:35:45.447397 E mitogen: attr: '__path__': _NamespacePath(['/nix/store/fyskdqxrb77qwzfldwmxgs99xwf6lp2q-python3.11-configparser-6.0.0/lib/python3.11/site-packages/backports'])
ERROR! [mux 2268210] 22:35:45.447417 E mitogen: attr: '__spec__': ModuleSpec(name='backports', loader=<_frozen_importlib_external.NamespaceLoader object at 0x7f5dd79bb290>, submodule_search_locations=_NamespacePath(['/nix/store/fyskdqxrb77qwzfldwmxgs99xwf6lp2q-python3.11-configparser-6.0.0/lib/python3.11/site-packages/backports']))
ERROR! [mux 2268210] 22:35:45.447441 E mitogen: attr: 'configparser': <module 'backports.configparser' from '/nix/store/fyskdqxrb77qwzfldwmxgs99xwf6lp2q-python3.11-configparser-6.0.0/lib/python3.11/site-packages/backports/configparser/__init__.py'>
As you can see, the backports module has its __file__ set to None. Providing None to os.path.abspath() causes the exception,
TypeError: expected str, bytes or os.PathLike object, not NoneType
Here's the fix that works for us: https://github.com/mitogen-hq/mitogen/pull/1042 Which is essentially
diff --git a/mitogen/master.py b/mitogen/master.py
index 4fb535f0a..73c8e4607 100644
--- a/mitogen/master.py
+++ b/mitogen/master.py
@@ -137,7 +137,11 @@ def is_stdlib_name(modname):
return False
# six installs crap with no __file__
- modpath = os.path.abspath(getattr(module, '__file__', ''))
+ path = getattr(module, '__file__', '')
+ if path is None:
+ path = ''
+
+ modpath = os.path.abspath(path)
return is_stdlib_path(modpath)
