saltstack
3006.6 unresponsive minions
Description
Every time there is a new Salt release, I run a salt command to upgrade all my servers:
salt '*' pkg.list_upgrades
About 1/2 the servers fail to get back to a state where the salt minion connects back to the controller.
Then I have to manually log into each server and restart the salt minion.
Please can this be tested and improved.
Setup
- [x] classic packaging
- [x] onedir packaging
Debian 12, with packages from the Salt repo.
@SwimGeek Can you please try setting request_channel_timeout: 60 in your minions' config? I believe that timeout un-intentionally got changed from 60 to 30.
Hi
Did a web search for 'request_channel_timeout' and searched through some config files. Did not find much.
Added to config and restarted. Seems to work.
Where do I read about this config option?
Hi
Did a web search for 'request_channel_timeout' and searched through some config files. Did not find much.
Added to config and restarted. Seems to work.
Where do I read about this config option?
https://docs.saltproject.io/en/latest/ref/configuration/minion.html#request-channel-timeout
Thanks. Will test this when the next round of Salt upgrades are available.
Hi, I tested by upgrading to 3006.7.
Upgraded 4 minions, all had minion config with 'request_channel_timeout: 60'
They did not connect to the controller after I sent upgrade with: salt '*' pkg.upgrade
Minion logs show:
2024-02-22 18:35:03,556 [salt.utils.parsers:1065][WARNING ][529109] Minion received a SIGTERM. Exiting.
if a salt-minion is one of the things in a pkg.upgrade it normally is not a good idea to do a pkg.upgrade. it will break the upgrade partway through the upgrade because the minion running the upgrade will be stopped.
upgrading a minion should be separate. something along the lines of salt <minion> cmd.run_bg 'salt-call --local pkg.install salt-minion' this command will separate the call from the running minion daemon, you won't get back a response. but will keep the minion from breaking cause it is trying to upgrade a running process which it stops.