saltstack
[BUG] 3007.1 - salt-proxy - "there is no current event loop in thread"
Description attempt to run a state.apply against a salt-sproxy device.
Setup (Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)
Please be as specific as possible and give set-up details.
- [ ] on-prem machine
- [X] VM (Virtualbox, KVM, etc. please specify)
- [ ] VM running on a cloud service, please be explicit and add details
- [ ] container (Kubernetes, Docker, containerd, etc. please specify)
- [ ] or a combination, please be explicit
- [ ] jails if it is FreeBSD
- [ ] classic packaging
- [ ] onedir packaging
- [ ] used bootstrap to install
Steps to Reproduce the behavior setup a salt-sproxy host + master setup. both version 3007.1 (in my case i was using the napalm nxos sproxy config) confirm that basic commands like test.ping work to the salt-proxy host. when you attempt to apply the following error occurs:
salt "sproxyminionname" state.apply timeout=600 test=true
sproxyminionname:
The minion function caused an exception: Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/metaproxy/proxy.py", line 483, in thread_return
return_data = minion_instance.executors[fname](
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in __call__
ret = self.loader.run(run_func, *args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
return _func_or_method(*args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/executors/direct_call.py", line 10, in execute
return func(*args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 160, in __call__
ret = self.loader.run(run_func, *args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1269, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1284, in _run_as
return _func_or_method(*args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/state.py", line 835, in apply_
return highstate(**kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/state.py", line 1154, in highstate
st_ = salt.state.HighState(
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 5007, in __init__
self.client = salt.fileclient.get_file_client(self.opts)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 55, in get_file_client
return {"remote": RemoteClient, "local": FSClient, "pillar": PillarClient}.get(
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1129, in __init__
self.channel = salt.channel.client.ReqChannel.factory(self.opts)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 56, in factory
return SyncWrapper(
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 76, in __init__
self.obj = cls(*args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 134, in factory
auth = salt.crypt.AsyncAuth(opts, io_loop=io_loop)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 695, in __new__
auth.__singleton_init__(opts, io_loop=io_loop)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/crypt.py", line 742, in __singleton_init__
self._authenticate_future = tornado.concurrent.Future()
File "/opt/saltstack/salt/lib/python3.10/asyncio/events.py", line 656, in get_event_loop
raise RuntimeError('There is no current event loop in thread %r.'
RuntimeError: There is no current event loop in thread 'ProcessPayload(jid=20240801160004816387)'.
ERROR: Minions returned with non-zero exit code
Expected behavior state to apply and respond cleanly
Screenshots If applicable, add screenshots to help explain your problem.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)Salt Version:
Salt: 3007.1
Python Version:
Python: 3.10.14 (main, Apr 3 2024, 21:30:09) [GCC 11.2.0]
Dependency Versions:
cffi: 1.16.0
cherrypy: unknown
dateutil: 2.8.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.4
libgit2: 1.7.1
looseversion: 1.3.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.7
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 23.1
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: 1.13.1
python-gnupg: 0.5.2
PyYAML: 6.0.1
PyZMQ: 25.1.2
relenv: 0.16.0
smmap: Not Installed
timelib: 0.3.0
Tornado: 6.3.3
ZMQ: 4.3.4
Salt Package Information:
Package Type: onedir
System Versions:
dist: debian 11.10 bullseye
locale: utf-8
machine: x86_64
release: 5.10.0-30-amd64
system: Linux
version: Debian GNU/Linux 11.10 bullseye
Additional context downgrading the salt-minion (and thus salt-sproxy) to 3006.9 fixes and state.apply's from the salt 3007.1 master work correctly
any update here ? i see a similar issue if i try to use salt.client.Caller() within a thread, looks like a general issue with threading and salt 3007.x. This worked with 3006.x.
We've just hit more or less the same issue with __slot__s and the fileclient. Applying this patch seems to fix it
diff --git salt/utils/asynchronous.py salt/utils/asynchronous.py
index 316fc5c478d..2db45ec49bc 100644
--- salt/utils/asynchronous.py
+++ salt/utils/asynchronous.py
@@ -73,7 +73,8 @@ class SyncWrapper:
self.cls = cls
if loop_kwarg:
kwargs[self.loop_kwarg] = self.io_loop
- self.obj = cls(*args, **kwargs)
+ with current_ioloop(self.io_loop):
+ self.obj = cls(*args, **kwargs)
self._async_methods = list(
set(async_methods + getattr(self.obj, "async_methods", []))
)
^can confirm that diff also fixes my issue with salt.client.Caller() within a thread.
@frebib would you consider opening a PR for that or would you be ok with me opening one with that change?
