saltstack
[BUG] config.items works via salt-call but throws exception when executed via master
Description
When config.items is executed from the master on a minion, it doesn't return anything:
root@test:~# salt test config.items
test:
Minion did not return. [No response]
The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:
salt-run jobs.lookup_jid 20230921131944857796
ERROR: Minions returned with non-zero exit code
But it produces this exception in the minion log:
2023-09-21 13:19:44,960 [salt.utils.process:998 ][ERROR ][2811] An un-handled exception from the multiprocessing process 'ProcessPayload(jid=20230921131944857796)' was caught:
Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/process.py", line 993, in wrapped_run_func
return run_func()
File "/opt/saltstack/salt/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 1844, in _target
run_func(minion_instance, opts, data)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 1838, in run_func
return Minion._thread_return(minion_instance, opts, data)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 2074, in _thread_return
minion_instance._return_pub(ret)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 2276, in _return_pub
ret_val = self._send_req_sync(load, timeout=timeout)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/minion.py", line 1609, in _send_req_sync
return event.fire_event(
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py", line 815, in fire_event
dump_data = salt.payload.dumps(data, use_bin_type=True)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/payload.py", line 173, in dumps
return salt.utils.msgpack.packb(
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/msgpack.py", line 133, in packb
return msgpack.packb(o, **_sanitize_msgpack_kwargs(kwargs))
File "/opt/saltstack/salt/lib/python3.10/site-packages/msgpack/__init__.py", line 35, in packb
return Packer(**kwargs).pack(o)
File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 298, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 295, in msgpack._cmsgpack.Packer.pack
File "msgpack/_packer.pyx", line 231, in msgpack._cmsgpack.Packer._pack
File "msgpack/_packer.pyx", line 285, in msgpack._cmsgpack.Packer._pack
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/payload.py", line 168, in ext_type_encoder
return dict(obj)
File "/opt/saltstack/salt/lib/python3.10/_collections_abc.py", line 886, in __iter__
yield from self._mapping
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/context.py", line 97, in __iter__
return self.value().__iter__()
AttributeError: 'NoneType' object has no attribute '__iter__'
But executing salt-call config.items locally on the minion, it works and returns the dict with all config parameters
It doesn't matter if it is the local minion (on the master-machine) or if it is a different/remote minion.
Setup simple test setup: one LXC container, Ubuntu Jammy with salt-master and salt-minion (packages from the salt-repo), default config
- [X] VM (Virtualbox, KVM, etc. please specify)
- [X] container (LXC)
Steps to Reproduce the behavior
run salt '*' config.items against any minion
Versions Report
salt --versions-report
Salt Version:
Salt: 3006.3
Python Version:
Python: 3.10.13 (main, Sep 6 2023, 02:11:27) [GCC 11.2.0]
Dependency Versions:
cffi: 1.14.6
cherrypy: unknown
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.2
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.9.8
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.13.10
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: ubuntu 22.04.3 jammy
locale: utf-8
machine: x86_64
release: 5.4.0-162-generic
system: Linux
version: Ubuntu 22.04.3 jammy
My humble impression is that Salt code is getting overcomplicated beyond maintainability. At least any major upgrade is an adventure and we've used salt since version 2016.3 or even 2015.something and have upgraded a few times.
I can see that NamedLoaderContext was introduced by @dwoz in https://github.com/saltstack/salt/commit/83f551aeca1552a57044416a14c6095091127bfb, it works for config.get but by the time salt gets round to packing it into zmq its self.value() gets None and __iter__() fails.
If congfig.items does this instead of returning opts directly:
result = {}
for opt, val in __opts__.items():
result[opt] = val
return val
then it magically works. This opt NamedLoaderContext object just gets messed up later in the process of returning job result and as said above its self.value() is None and it's useless.
@szjur
My humble impression is that Salt code is getting overcomplicated beyond maintainability. At least any major upgrade is an adventure and we've used salt since version 2016.3 or even 2015.something and have upgraded a few times.
Salt became un-maintainable long ago. The core team recognized this and have been working extremely hard for quite some time now to correct course. Off the top of my head some things we've done:
- Started requiring tests for changes. It's hard to believe that a basic function like config.items didn't have sufficient test coverage to catch this bug. What is even harder to believe is that we still have contributors (who are also users) that complain about this requirement.
- Started requiring a fully passing test suite before any release.
- We've fully automated 99% of what's required to do a release which took our release time down from weeks (in some cases) to about an hour. This has greatly increased our ability to iterate and fix problems. I think 3006.x has been a testament to this improvement.
- Simplified our dependency tree with onedir packaging. I'll be the first to admit this has also come with it's share of problems but things are vastly better now than in previous releases and we are still working to correct any remaining issues.
Salt is not only maintainable but is being maintained and improved.
