salt
salt copied to clipboard
Salt API crashes if Python is updated
Description of Issue
Last night our server running Salt (master, api, and a minion for the master) upgraded Python via unattended-upgrades. This caused the salt-api
process to crash. It appears that the process was running, but become unresponsive to HTTP requests.
This was in the system log files:
Dec 16 06:29:35 saltstack-master.internal.fusionauth.io salt-api[302948]: [16/Dec/2021:06:29:35] ENGINE Restarting because /usr/lib/python3.8/email/base64mime.py changed.
Dec 16 06:29:35 saltstack-master.internal.fusionauth.io salt-api[302948]: [16/Dec/2021:06:29:35] ENGINE Stopped thread 'Autoreloader'.
Dec 16 06:29:35 saltstack-master.internal.fusionauth.io salt-api[302948]: [16/Dec/2021:06:29:35] ENGINE Bus STOPPING
Additionally, here is the Python update in the APT logs:
Start-Date: 2021-12-16 06:29:33
Commandline: /usr/bin/unattended-upgrade
Upgrade: libpython3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04.1, 3.8.10-0ubuntu1~20.04.2), libpython3.8:amd64 (3.8.10-0ubuntu1~20.04.1, 3.8.10-0ubuntu1~20.04.2), python3.8:amd64 (3.8.10-0ubuntu1~20.04.1, 3.8.10-0ubuntu1~20.04.2), python3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04.1, 3.8.10-0ubuntu1~20.04.2), libpython3.8-stdlib:amd64 (3.8.10-0ubuntu1~20.04.1, 3.8.10-0ubuntu1~20.04.2)
End-Date: 2021-12-16 06:29:36
Setup
Here's our master config:
rest_cherrypy:
port: 8080
thread_pool: 10
socket_queue_size: 5
disable_ssl: True
debug: True
external_auth:
file:
^filename: /etc/salt/users.txt
<redacted>:
- .*
<redacted>:
- test.ping
Versions Report
Salt Version:
Salt: 3003
Dependency Versions:
cffi: Not Installed
cherrypy: unknown
dateutil: 2.7.3
docker-py: Not Installed
gitdb: 2.0.6
gitpython: 3.0.7
Jinja2: 2.10.1
libgit2: 0.28.3
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: Not Installed
pycryptodome: 3.6.1
pygit2: 1.0.3
Python: 3.8.10 (default, Nov 26 2021, 20:14:08)
python-gnupg: 0.4.5
PyYAML: 5.3.1
PyZMQ: 18.1.1
smmap: 2.0.5
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.2
System Versions:
dist: ubuntu 20.04 focal
locale: utf-8
machine: x86_64
release: 5.4.0-1047-aws
system: Linux
version: Ubuntu 20.04 focal
Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:
- Community Wiki
- Salt’s Contributor Guide
- Join our Community Slack
- IRC on LiberaChat
- Salt Project YouTube channel
- Salt Project Twitch channel
There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!
@voidmain Thanks for the report. The issue above seems related to cherrypy, which is causing the restart when the underlying Python files changed. With Salt moving to onedir packages, https://docs.saltproject.io/en/master/topics/releases/3005.html#new-packages-available, and no longer relying on the system version of Python this particular issue should no longer occur.
Looks like Onedir is broken on ARM64 Linux. I opened another issue (#62757) for that because we are using Graviton EC2 instances for everything and can't upgrade to Onedir until it is fixed.
Closing this due to age and the old version of Salt. Can you retest this with the latest release of Salt 3006.2 and if this is still an issue, please open a new issue.