salt
salt copied to clipboard
[BUG] Salt Exception with message "Event iteration failed with exception: 'list' object has no attribute 'items'"
Description We did not change any configuration, so the setup was running before. We want to deploy a Mailserver using Terraform which is calling the salt stack.
When running a High State, we get the following errors:
Minion Log:
module.product_mailservers[3].module.nutanixvm.null_resource.salt-bootstrap (remote-exec): [DEBUG ] LazyLoaded highstate.output
module.product_mailservers[3].module.nutanixvm.null_resource.salt-bootstrap (remote-exec): local:
module.product_mailservers[3].module.nutanixvm.null_resource.salt-bootstrap (remote-exec): Data failed to compile:
module.product_mailservers[3].module.nutanixvm.null_resource.salt-bootstrap (remote-exec): ----------
module.product_mailservers[3].module.nutanixvm.null_resource.salt-bootstrap (remote-exec): Pillar failed to render with the following messages:
module.product_mailservers[3].module.nutanixvm.null_resource.salt-bootstrap (remote-exec): ----------
module.product_mailservers[3].module.nutanixvm.null_resource.salt-bootstrap (remote-exec): Rendering SLS 'mailserver.opendkim' failed. Please see master log for details.
module.product_mailservers[3].module.nutanixvm.null_resource.salt-bootstrap (remote-exec): ----------
module.product_mailservers[3].module.nutanixvm.null_resource.salt-bootstrap (remote-exec): Rendering SLS 'mailserver.opendkim' failed. Please see master log for details.
Master Log:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/salt/pillar/__init__.py", line 896, in render_pstate
**defaults
File "/usr/lib/python3/dist-packages/salt/template.py", line 99, in compile_template
ret = render(input_data, saltenv, sls, **render_kwargs)
File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 149, in __call__
return self.loader.run(run_func, *args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1201, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1216, in _run_as
return _func_or_method(*args, **kwargs)
File "/usr/lib/python3/dist-packages/salt/renderers/jinja.py", line 75, in render
**kws
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 261, in render_tmpl
output = render_str(tmplstr, context, tmplpath)
File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 509, in render_jinja_tmpl
raise SaltRenderError("Jinja variable {}{}".format(exc, out), buf=tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'dict object' has no attribute 'addresses'
2023-06-27 15:16:45,724 [salt.pillar :1224][CRITICAL][134604] Pillar render error: Rendering SLS 'mailserver.opendkim' failed. Please see master log for details.
2023-06-27 15:16:45,724 [salt.pillar :1224][CRITICAL][134604] Pillar render error: Rendering SLS 'mailserver.opendkim' failed. Please see master log for details.
2023-06-27 15:16:45,883 [salt.utils.event :908 ][ERROR ][134459] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-06-27 15:16:55,288 [salt.utils.event :908 ][ERROR ][134475] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-06-27 15:18:18,594 [salt.utils.event :908 ][ERROR ][134515] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-06-27 15:23:25,128 [salt.utils.event :908 ][ERROR ][134491] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-06-27 15:24:07,291 [salt.utils.event :908 ][ERROR ][134514] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-06-27 15:27:16,587 [salt.utils.event :908 ][ERROR ][134521] Event iteration failed with exception: 'list' object has no attribute 'items'
Setup
- [ X] on-prem machine
- [ X] VM - Nutanix
- [ X] in combination with Terraform - TF calls the salt stack do deploy an application
The following file is called when the problem occurs:
opendkim.sls
:
{%- if pillar.get('opendkim-key') %}
include:
- base: opendkim
opendkim-schedule:
schedule.present:
- function: state.sls
- job_args:
- mailserver.opendkim
- seconds: 21600
- splay:
start: 300
end: 1500
- run_on_start: false
- maxrunning: 1
- jid_include: True
{%- else %}
opendkim-schedule:
schedule.absent:
- name: opendkim-schedule
{%- endif %}
Steps to Reproduce the behavior Run the state either manually or use Terraform.
Expected behavior Salt Stack should be called, running and deploy the Mailserver as configured.
Screenshots I don't think they're needed here.
Versions Report
root@salt01: /var/log/salt # salt --versions-report
Salt Version:
Salt: 3004.2
Dependency Versions:
cffi: 1.15.1
cherrypy: Not Installed
dateutil: 2.7.3
docker-py: Not Installed
gitdb: 2.0.5
gitpython: 2.1.11
Jinja2: 2.10
libgit2: 1.5.0
M2Crypto: Not Installed
Mako: 1.1.3
msgpack: 0.5.6
msgpack-pure: Not Installed
mysql-python: 1.3.10
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.6.1
pygit2: 1.10.0
Python: 3.7.3 (default, Jan 22 2021, 20:04:44)
python-gnupg: Not Installed
PyYAML: 3.13
PyZMQ: 17.1.2
smmap: 2.0.5
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.1
System Versions:
dist: debian 10 buster
locale: UTF-8
machine: x86_64
release: 4.19.0-21-amd64
system: Linux
version: Debian GNU/Linux 10 buster
Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:
- Community Wiki
- Salt’s Contributor Guide
- Join our Community Slack
- IRC on LiberaChat
- Salt Project YouTube channel
- Salt Project Twitch channel
There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!
What is trying to access that addresses
key?
May have similar, see Salt Community Slack , general channel https://saltstackcommunity.slack.com/archives/C7K04SEJC/p1705011335140729
here it is in case contributor not on Slack
kmm
3:15 PM
In 3006.5 (running on the salt master and minions), I'm running into many occurrences of the error [salt.utils.event :912 ][ERROR ][2758413] Event iteration failed with exception: 'list' object has no attribute 'items' in the salt master's log file. These errors are immediately preceded by an event in the form:
[salt.utils.event :823 ][DEBUG ][2758413] Sending event: tag = salt/job/[jid]/ret/[name]; data = {'cmd': '_return', 'id': '[minion_id]', 'fun': 'state.sls', 'fun_args': ['arg1', 'arg2'], 'schedule': '[scheduler state]', 'jid': '[jid]', 'pid': 656, 'return': ['The function "state.sls" is running as PID 4252 and was started at 2024, Jan 11 22:09:45.272142 with jid [jid]'], 'retcode': 1, 'success': True, '_stamp': '2024-01-11T22:09:49.164465', 'out': 'highstate', 'arg': ['arg1', 'arg2'], 'tgt_type': 'glob', 'tgt': '[minion_id]'}
However, in most other "Sending event:" log messages, the return key has a dictionary value, rather than a list/str (which is what's causing the error). Has anyone previously encountered this before?
@dmurphy18 I'll provide a little more context here about the issue. In one instance, the "Event iteration failed..." was generated as a result of this error in a minion:
2024-01-11 14:35:53,434 [salt.utils.schedule:913 ][ERROR ][2592540] Unhandled exception running state.sls
Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/schedule.py", line 880, in handle_func
ret["return"] = self.functions[func](*args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
ret = self.loader.run(run_func, *args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
return _func_or_method(*args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/state.py", line 1357, in sls
conflict = _check_queue(queue, kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/state.py", line 430, in _check_queue
conflict = running(concurrent=kwargs.get("concurrent", False))
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/state.py", line 381, in running
active = __salt__["saltutil.is_running"]("state.*")
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/context.py", line 78, in __getitem__
return self.value()[item]
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 349, in __getitem__
super().__getitem__(item) # try to get the item from the dictionary
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/lazy.py", line 105, in __getitem__
raise KeyError(key)
KeyError: 'saltutil.is_running'
That error in the minion resulted in the following error in the salt master's log file:
2024-01-11 14:32:53,449 [salt.utils.event :823 ][DEBUG ][2758412] Sending event: tag = salt/job/[jid]/ret/[minion_id]; data = {'cmd': '_return', 'id': '[minion_id]', 'fun': 'state.sls', 'fun_args': ['arg1'], 'schedule': 'schedule1', 'jid': '[jid]', 'pid': 2592426, 'return': 'Unhandled exception running state.sls', 'success': False, 'retcode': 254, '_stamp': '2024-01-11T22:32:53.449264', 'out': 'highstate', 'arg': ['arg1'], 'tgt_type': 'glob', 'tgt': '[minion_id]'}
2024-01-11 14:32:53,450 [salt.utils.event :912 ][ERROR ][2758412] Event iteration failed with exception: 'str' object has no attribute 'items'
In another instance, the log message in the salt-master was:
[salt.utils.event :823 ][DEBUG ][2758403] Sending event: tag = salt/job/[jid]/ret/[minion_id]; data = {'cmd': '_return', 'id': '[minion_id]', 'fun': 'state.sls', 'fun_args': ['arg1'], 'schedule': 'schedule2', 'jid': '[jid]', 'pid': 2593155, 'return': ["Detected conflicting IDs, SLS IDs need to be globally unique.\n The conflicting ID is 'handle_unsupported_system' and is found in SLS 'base:[salt state1]' and SLS 'base:[salt state2]'"], 'retcode': 1, 'success': True, '_stamp': '2024-01-11T22:42:35.930293', 'out': 'highstate', 'arg': ['arg1'], 'tgt_type': 'glob', 'tgt': '[minion_id]'}
However, for the vast majority of minions, the "Sending event" log message is accompanied by a data dictionary containing a return
key with a dictionary as its value. Even more perplexing, it appears restarting the affected minions resolves the issue, but only until the next time the salt master becomes overloaded & causes minion timeouts and associated errors. It might be worth mentioning that these timeouts are frequently associated with the following error:
2024-01-11 22:31:48,936 [salt.utils.schedule:913 ][ERROR ][1779876] Unhandled exception running state.sls
Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1157, in _channel_send
return self.channel.send(
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 125, in wrap
raise exc_info[1].with_traceback(exc_info[2])
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 131, in _target
result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync
return future_cell[0].result()
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 341, in send
ret = yield self._crypted_transfer(load, timeout=timeout, raw=raw)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 294, in _crypted_transfer
ret = yield _do_transfer()
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 275, in _do_transfer
data = yield self.transport.send(
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", line 909, in send
ret = yield self.message_client.send(load, timeout=timeout)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw(*exc_info)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", line 589, in send
recv = yield future
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
salt.exceptions.SaltReqTimeoutError: Message timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/schedule.py", line 880, in handle_func
ret["return"] = self.functions[func](*args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
ret = self.loader.run(run_func, *args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
return _func_or_method(*args, **kwargs)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/state.py", line 1466, in sls
high_, errors = st_.render_highstate({opts["saltenv"]: mods})
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 4642, in render_highstate
statefiles = fnmatch.filter(self.avail[saltenv], sls_match)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 3734, in __getitem__
self._avail[saltenv] = self._hs.client.list_states(saltenv)
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 385, in list_states
for path in self.file_list(saltenv):
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1376, in file_list
return self._channel_send(
File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1162, in _channel_send
raise SaltClientError(
salt.exceptions.SaltClientError: File client timed out after 1691084572
Master version info:
[me@master]# salt-run --versions-report
Salt Version:
Salt: 3006.5
Python Version:
Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]
Dependency Versions:
cffi: 1.14.6
cherrypy: 18.6.1
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.2
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.9.8
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.14.2
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: rocky 8.9 Green Obsidian
locale: utf-8
machine: x86_64
release: 4.18.0-513.9.1.el8_9.x86_64
system: Linux
version: Rocky Linux 8.9 Green Obsidian
Minion version info:
[me@minion]# salt-call --versions-report
Salt Version:
Salt: 3006.5
Python Version:
Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]
Dependency Versions:
cffi: 1.14.6
cherrypy: 18.6.1
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.2
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.9.8
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.14.2
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: ubuntu 20.04.6 focal
locale: utf-8
machine: x86_64
release: 5.15.0-1038-gcp
system: Linux
version: Ubuntu 20.04.6 focal
I'm also seeing this message, but as mentioned it's only happening as a side effect when a state run fails otherwise. I'm reasonably sure it's caused by:
https://github.com/saltstack/salt/blob/daef7613bc359822a4f04ffdd7a91a9edcac7c80/salt/utils/event.py#L823-L829
For fatal errors, the state
module usually returns lists (rendering errors)/strings (uncaught exceptions), which end up in return
.
@lkubb Even for informational messages, the state
module can return lists, e.g.:
[me@salt-master]# salt-run state.event pretty=True
[...]
salt/job/[jid]/ret/[minion_id] {
"_stamp": "2024-01-12T00:33:39.215860",
"arg": [
"[arg1]"
],
"cmd": "_return",
"fun": "state.sls",
"fun_args": [
"[arg1]"
],
"id": "[minion_id]",
"jid": "[jid]",
"out": "highstate",
"pid": 12270,
"retcode": 1,
"return": [
"The function \"state.sls\" is running as PID 12250 and was started at 2024, Jan 12 00:33:36.053637 with jid [jid]"
],
"schedule": "[schedule1]",
"success": true,
"tgt": "[minion_id]",
"tgt_type": "glob"
}
[...]
The salt master's log file shows the following at the same time (in a different time zone):
2024-01-11 16:33:39,216 [salt.utils.event :912 ][ERROR ][2229253] Event iteration failed with exception: 'list' object has no attribute 'items'
Yeah, in my mind I had classified this as a rendering error since no states ran. I wouldn't call it informational though, it's failing the requested execution for good.
Either way, the linked code should handle lists and strings in return
as well, which would likely fix this issue.
looks like this is a side effect of #60782 which added the error message versus the "pass" that had been in the exception block for ages (back to at least v2016.11) in the _fire_ret_load_specific_fun
function
we're using the following patch to suppress the error again:
--- /opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py 2023-10-16 12:22:41.000000000 -0500
+++ /tmp/event.py 2024-01-24 09:08:48.923528695 -0600
@@ -890,24 +890,25 @@
retcode = load["retcode"]
try:
- for tag, data in ret.items():
- data["retcode"] = retcode
- tags = tag.split("_|-")
- if data.get("result") is False:
- self.fire_event(
- data, "{}.{}".format(tags[0], tags[-1])
- ) # old dup event
- data["jid"] = load["jid"]
- data["id"] = load["id"]
- data["success"] = False
- data["return"] = "Error: {}.{}".format(tags[0], tags[-1])
- data["fun"] = fun
- if "user" in load:
- data["user"] = load["user"]
- self.fire_event(
- data,
- tagify([load["jid"], "sub", load["id"], "error", fun], "job"),
- )
+ if getattr(ret, "items", None):
+ for tag, data in ret.items():
+ data["retcode"] = retcode
+ tags = tag.split("_|-")
+ if data.get("result") is False:
+ self.fire_event(
+ data, "{}.{}".format(tags[0], tags[-1])
+ ) # old dup event
+ data["jid"] = load["jid"]
+ data["id"] = load["id"]
+ data["success"] = False
+ data["return"] = "Error: {}.{}".format(tags[0], tags[-1])
+ data["fun"] = fun
+ if "user" in load:
+ data["user"] = load["user"]
+ self.fire_event(
+ data,
+ tagify([load["jid"], "sub", load["id"], "error", fun], "job"),
+ )
except Exception as exc: # pylint: disable=broad-except
log.error(
"Event iteration failed with exception: %s",
likely should be handled differently, as @lkubb described, to still fire an event on the errors that are returned as lists/strings/etc, but from our perspective, removing the log "spam" and returning to the previous state before #60782 were good enough for us at this time
@morenski
File client timed out after 1691084572
indicates that the fileclient has timed out. You can adjust this timeout with the request_channel_timeout
and request_channel_tries
minion config option.