salt icon indicating copy to clipboard operation
salt copied to clipboard

[BUG] `beacon.present` state TypeError traceback

Open lkubb opened this issue 9 months ago • 0 comments

Description Sometimes - I assume when the minion-master connection is temporarily unavailable * - beacon.present just crashes with a traceback: TypeError: argument of type 'NoneType' is not iterable

Setup (Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

  • [x] on-prem machine
  • [x] VM (KVM)
  • [x] onedir packaging
  • [x] used bootstrap to install

* The behavior usually manifests shortly after a daemon restart - startup_states: highstate is in my config. Maybe there is some race condition.

Steps to Reproduce the behavior

  • stop the local Salt minion
  • run salt-call --local state.single beacon.present

Expected behavior Return an error description or retry (it happens when the daemon is up, the reproduction above just simulates what happens).

Screenshots

[ERROR   ] Encountered StreamClosedException
[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 2424, in call
    ret = self.states[cdata["full"]](
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1251, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1266, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1299, in wrapper
    return f(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/states/beacon.py", line 104, in present
    if name in current_beacons:
TypeError: argument of type 'NoneType' is not iterable

local:
----------
          ID: test
    Function: beacon.present
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 2424, in call
                  ret = self.states[cdata["full"]](
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
                  ret = self.loader.run(run_func, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1251, in run
                  return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1266, in _run_as
                  return _func_or_method(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1299, in wrapper
                  return f(*args, **kwargs)
                File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/states/beacon.py", line 104, in present
                  if name in current_beacons:
              TypeError: argument of type 'NoneType' is not iterable
     Started: 21:22:55.647197
    Duration: 1009.19 ms
     Changes:

Summary for local
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:   1.009 s

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:
          Salt: 3006.7

Python Version:
        Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.11
     gitpython: 3.1.42
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.15.1
         smmap: 5.0.1
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: rocky 9.3 Blue Onyx
        locale: utf-8
       machine: x86_64
       release: 5.14.0-362.13.1.el9_3.x86_64
        system: Linux
       version: Rocky Linux 9.3 Blue Onyx

Additional context This is caused by beacons.list returning None when event.fire fails and beacon.present indiscriminately iterating over the result.

I think the solution should be twofold:

  • check if the return value is None
  • if so, retry after a delay up to a configurable timeout

Another note: beacons.list potentially returns three different value types:

  1. None when the event fails completely
  2. a ret dict when unexpected data is returned
  3. the actual list of beacons

2. could cause weird behavior as well.

lkubb avatar Apr 25 '24 21:04 lkubb