[DOCS] solution in FAQ on restarting minion after upgrade no longer works

[bensteinberg]

Description The policy layer method in https://docs.saltproject.io/en/latest/faq.html#upgrade-without-automatic-restart which used to work correctly, no longer works: applying this state now results in the following trace:

    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python3/dist-packages/salt/minion.py", line 1905, in _thread_return
        for executor in executors
      File "/usr/lib/python3/dist-packages/salt/minion.py", line 1861, in _execute_job_function
        executors[-1] = "sudo"  # replace the last one with sudo
      File "/usr/lib/python3/dist-packages/salt/loader.py", line 1241, in __call__
      File "/usr/lib/python3/dist-packages/salt/loader.py", line 2274, in run
      File "/usr/lib/python3/dist-packages/salt/loader.py", line 2289, in _run_as
      File "/usr/lib/python3/dist-packages/salt/executors/direct_call.py", line 12, in execute
      File "/usr/lib/python3/dist-packages/salt/loader.py", line 1241, in __call__
      File "/usr/lib/python3/dist-packages/salt/loader.py", line 2274, in run
      File "/usr/lib/python3/dist-packages/salt/loader.py", line 2289, in _run_as
      File "/usr/lib/python3/dist-packages/salt/modules/state.py", line 792, in apply_
        if mods:
      File "/usr/lib/python3/dist-packages/salt/modules/state.py", line 1391, in sls
        snapper_pre = _snapper_pre(opts, kwargs.get("__pub_jid", "called localy"))
      File "/usr/lib/python3/dist-packages/salt/state.py", line 3269, in call_high
        # Verify that the high data is structurally sound
      File "/usr/lib/python3/dist-packages/salt/state.py", line 2495, in call_chunks
        if action == "kill":
      File "/usr/lib/python3/dist-packages/salt/state.py", line 2980, in call_chunk
        self.event(
      File "/usr/lib/python3/dist-packages/salt/state.py", line 2999, in call_chunk
        elif status == "met":
      File "/usr/lib/python3/dist-packages/salt/utils/decorators/state.py", line 45, in _func
        result = func(*args, **kwargs)
      File "/usr/lib/python3/dist-packages/salt/state.py", line 2229, in call
        if low.get("__prereq__"):
      File "/usr/lib/python3/dist-packages/salt/state.py", line 1256, in check_refresh
        return
      File "/usr/lib/python3/dist-packages/salt/state.py", line 1213, in module_refresh
        Refresh all the modules
      File "/usr/lib/python3/dist-packages/salt/state.py", line 1191, in load_modules
        providers = {}
      File "/usr/lib/python3/dist-packages/salt/loader.py", line 738, in render
    salt.exceptions.LoaderError: The renderer jinja|yaml is unavailable, this error is often because the needed software is unavailable

Type of documentation FAQ -- though this perhaps should be part of core documentation.

Location or format of documentation https://docs.saltproject.io/en/latest/faq.html#what-is-the-best-way-to-restart-a-salt-minion-daemon-using-salt-after-upgrade

[bensteinberg]

I should have mentioned, this is on the upgrade from 3003.3+ds-1 to 3004+ds-1 -- all the machines in question are running Debian buster.

[koebbe]

Same here. Could it have something to do with the fact that the directory structure changed from 3003 to 3004? For example:

3003 - /usr/lib/python3/dist-packages/salt/loader.py   (file)
3004 - /usr/lib/python3/dist-packages/salt/loader/     (directory)

[bensteinberg]

I should also have mentioned, that as it stands, the failed state leaves in place the file /usr/sbin/policy-rc.d with contents

#!/bin/sh
exit 101

[smarsching]

I used the following state as a workaround to update the minions:

upgrade_minion:
  cmd.run:
    - name: 'apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --only-upgrade salt-common salt-minion'
    - bg: True

It is not very elegant and obviously it will only work for Debian-based systems, but it is still better than having to do the upgrade manually.

[bensteinberg]

This method is working again, on the upgrade from 3004+ds-1 to 3004.1+ds-1, of a mixed set of Debian buster and bullseye minions. I'll leave this open for now, in case you need to investigate further.

I believe #61462 is the same problem, and could similarly have been resolved.

[cp-samantha]

Interesting that this is resolved for you since I just attempted the 3004.1 upgrade and it still failed for me with the same jinja|yaml error it always fails on. :pensive:

[bensteinberg]

@cp-samantha For what it's worth, I upgraded the salt master first, according to https://docs.saltproject.io/en/latest/topics/releases/3004.1.html ("3004.1 minions are not able to communicate with masters older than 3004.1. You must upgrade your masters before upgrading minions.") -- I did that "by hand", with sudo apt upgrade, then was able to use the policy layer method to upgrade the minions.

[cp-samantha]

Silly me. I saw that note, but in all my swapping around between production salts and lab salts, I missed the upgrade in my lab env before testing the upgrade to a lab minion, which is where I've been testing this issue.

I managed to pull the deb for 3004 out of my apt cache and retry the upgrade after upgrading the master and it does look like the upgrade went off without a hitch when taking into consideration that the master should be updated first.

Thanks for the sanity check :smile:

[bensteinberg]

This method is not working again for the upgrade to 3005, using the onedir packaging. A simple salt <minionid> pkg.upgrade attempts but fails to start the salt minion:

Aug 29 13:30:07 ip-172-31-53-89 salt-minion[36307]: [INFO    ] An instance is already running. Exiting the Salt Minion
Aug 29 13:30:07 ip-172-31-53-89 salt-minion[36307]: [INFO    ] Shutting down the Salt Minion

Using the policy layer method gets

    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python3/dist-packages/salt/minion.py", line 1912, in _thread_return
      File "/usr/lib/python3/dist-packages/salt/minion.py", line 1869, in _execute_job_function
      File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 149, in __call__
      File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1201, in run
      File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1216, in _run_as
      File "/usr/lib/python3/dist-packages/salt/executors/direct_call.py", line 10, in execute
      File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 149, in __call__
      File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1201, in run
      File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1216, in _run_as
      File "/usr/lib/python3/dist-packages/salt/modules/state.py", line 793, in apply_
      File "/usr/lib/python3/dist-packages/salt/modules/state.py", line 1392, in sls
      File "/usr/lib/python3/dist-packages/salt/state.py", line 3279, in call_high
      File "/usr/lib/python3/dist-packages/salt/state.py", line 2497, in call_chunks
      File "/usr/lib/python3/dist-packages/salt/state.py", line 2984, in call_chunk
      File "/usr/lib/python3/dist-packages/salt/state.py", line 3003, in call_chunk
      File "/usr/lib/python3/dist-packages/salt/utils/decorators/state.py", line 45, in _func
      File "/usr/lib/python3/dist-packages/salt/state.py", line 2237, in call
      File "/usr/lib/python3/dist-packages/salt/state.py", line 1273, in check_refresh
      File "/usr/lib/python3/dist-packages/salt/state.py", line 1230, in module_refresh
      File "/usr/lib/python3/dist-packages/salt/state.py", line 1203, in load_modules
      File "/usr/lib/python3/dist-packages/salt/loader/__init__.py", line 755, in render
    salt.exceptions.LoaderError: The renderer jinja|yaml is unavailable, this error is often because the needed software is unavailable

and a stopped minion, but a manual restart of the minion at this point works fine. At the moment I don't have a way to upgrade minions without logging into them or being able to reboot them remotely.

(The policy layer method fails before deleting /usr/sbin/policy-rc.d, FWIW.)

[bensteinberg]

In some cases, the minion restarts some time after pkg.upgrade -- maybe a couple of minutes?

[barbaricyawps]

Can someone summarize for me whether the FAQ solution still works as documented? If not, could someone tell me which change needs to be made to the workaround documented in the FAQ?

I have a new MR open on the Salt Install Guide to incorporate the FAQ solution into the official docs (and other upgrade topic improvements) and I want to make sure it's accurate before merging, so feel free to comment on the merge if you have a GitLab account: https://gitlab.com/saltstack/open/docs/salt-install-guide/-/merge_requests/52

[barbaricyawps]

Okay, ha, one of my co-workers was eager to get this in and went ahead and merged in that merge request. Even though it's not open any more, I can still make any changes people would like in a new merge request.

[bensteinberg]

On release of 3005.1+ds-1, I'm still having trouble with the technique in the documentation; it leaves the minion not running and /usr/sbin/policy-rc.d still in place. A simple pkg.upgrade sometimes results in the minion restarting, but not always. I'm having better luck with this:

sudo salt myminion pkg.install at
sudo salt myminion at.at 'now + 1 minute' 'apt upgrade -y'

[alrf]

3006 - the same issue..

[smarsching]

@alrf Yes, the issue is still present in v3006. Luckily the workaround still works. Without using a state, it is even easier:

salt <target expression> cmd.run "apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --only-upgrade salt-minion" bg=True

I have used this method several times in different environments and it is very reliable.