saltstack
[BUG] salt-minion[xx]: [ERROR ] Publish server binding pub to /var/run/salt/minion/minion_event_c305107a7a_pub.ipc ssl=None
Description
After upgrade to 3007 and cleanup .pyc per https://github.com/saltstack/salt/issues/66177 still seeing this error message
salt-minion[xx]: [ERROR ] Publish server binding pub to /var/run/salt/minion/minion_event_c305107a7a_pub.ipc ssl=None
Setup (Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)
Please be as specific as possible and give set-up details.
- [x ] on-prem machine
- [x ] VM (Virtualbox, KVM, etc. please specify)
- [ x] onedir packaging
- [ x] used bootstrap to install to upgrade existing agent.
Steps to Reproduce the behavior
get and run bootstrap-salt.sh on existing system as root.
Expected behavior
bootstrap.sh will upgrade existing minion to latest version (3007 in this case) without error when starting up 3007 minion.
Screenshots [root@rocky8t03 ~]# systemctl status -l salt-minion ● salt-minion.service - The Salt Minion Loaded: loaded (/usr/lib/systemd/system/salt-minion.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2024-03-06 19:44:56 CST; 1min 28s ago Docs: man:salt-minion(1) file:///usr/share/doc/salt/html/contents.html https://docs.saltproject.io/en/latest/contents.html Main PID: 969955 (python3.10) Tasks: 11 (limit: 48605) Memory: 69.6M CGroup: /system.slice/salt-minion.service ├─969955 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion └─969963 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager MinionProcessManager
Mar 06 19:44:59 rocky8t03 salt-minion[969963]: c '/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/pycache/vault.cpython-310.pyc' and '/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/vault'
Mar 06 19:44:59 rocky8t03 salt-minion[969963]: [ERROR ] Module/package collision: '/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/pycache/vault.cpython-310.pyc' and '/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/vault'
Mar 06 19:44:59 rocky8t03 salt-minion[969963]: [ERROR ] Module/package collision: '/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/pycache/vault.cpython-310.pyc' and '/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/vault'
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:
Salt: 3007.0
Python Version:
Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]
Dependency Versions:
cffi: 1.16.0
cherrypy: unknown
dateutil: 2.8.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.3
libgit2: Not Installed
looseversion: 1.3.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.7
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 23.1
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.5.2
PyYAML: 6.0.1
PyZMQ: 25.1.2
relenv: 0.15.1
smmap: Not Installed
timelib: 0.3.0
Tornado: 6.3.3
ZMQ: 4.3.4
Salt Package Information:
Package Type: onedir
System Versions:
dist: rocky 8.9 Green Obsidian
locale: utf-8
machine: x86_64
release: 4.18.0-513.18.1.el8_9.x86_64
system: Linux
version: Rocky Linux 8.9 Green Obsidian
[tjyang@ilclsalt01 salt]$ salt --versions-report
Salt Version:
Salt: 3007.0
Python Version:
Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]
Dependency Versions:
cffi: 1.16.0
cherrypy: unknown
dateutil: 2.8.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.3
libgit2: Not Installed
looseversion: 1.3.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.7
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 23.1
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.5.2
PyYAML: 6.0.1
PyZMQ: 25.1.2
relenv: 0.15.1
smmap: Not Installed
timelib: 0.3.0
Tornado: 6.3.3
ZMQ: 4.3.4
Salt Package Information:
Package Type: onedir
System Versions:
dist: rocky 8.9 Green Obsidian
locale: utf-8
machine: x86_64
release: 4.18.0-513.18.1.el8_9.x86_64
system: Linux
version: Rocky Linux 8.9 Green Obsidian
Additional context current workaround is to remove /opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/pycache directory and restart salt-minion.
but still having following errors
CGroup: /system.slice/salt-minion.service
├─3442349 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion
└─3442363 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager MinionProce>
Mar 07 05:20:29 va32lsalt01 salt-minion[3442363]: File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", lin>
Mar 07 05:20:29 va32lsalt01 salt-minion[3442363]: await callback(decoded)
Mar 07 05:20:29 va32lsalt01 salt-minion[3442363]: TypeError: object NoneType can't be used in 'await' expression
Mar 07 05:20:34 va32lsalt01 salt-minion[3442363]: [ERROR ] Exception while running callback
Mar 07 05:20:34 va32lsalt01 salt-minion[3442363]: Traceback (most recent call last):
Mar 07 05:20:34 va32lsalt01 salt-minion[3442363]: File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", l>
Mar 07 05:20:34 va32lsalt01 salt-minion[3442363]: await callback(msg)
Mar 07 05:20:34 va32lsalt01 salt-minion[3442363]: File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", lin>
Mar 07 05:20:34 va32lsalt01 salt-minion[3442363]: await callback(decoded)
Mar 07 05:20:34 va32lsalt01 salt-minion[3442363]: TypeError: object NoneType can't be used in 'await' expression
I get similar error but on the master:
salt-master[2925533]: [ERROR ] Publish server binding pub to /var/run/salt/master/master_event_pub.ipc ssl=None
These look like they should be debug statements...
https://github.com/saltstack/salt/blob/562e739408ce4aef14400dc6aea91c12a3f5c425/salt/transport/tcp.py#L1404-L1420
@dwoz , can you confirm?
Master Description: After upgrade to 3007 and cleanup .pyc per https://github.com/saltstack/salt/issues/66177 seeing this error message:
systemctl status -l salt-master
salt-master.service - The Salt Master Server
Loaded: loaded (/lib/systemd/system/salt-master.service; enabled; preset: enabled)
Active: active (running) since Wed 2024-04-03 17:53:44 EDT; 8min ago
Docs: man:salt-master(1)
file:///usr/share/doc/salt/html/contents.html
https://docs.saltproject.io/en/latest/contents.html
Main PID: 1916 (/opt/saltstack/)
Tasks: 22 (limit: 4652)
Memory: 167.8M
CPU: 13.569s
CGroup: /system.slice/salt-master.service
├─1916 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master MainProcess"
├─2110 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master PubServerChannel._publish_daemon"
├─2111 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master EventPublisher"
├─2112 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master EventMonitor"
├─2115 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master Maintenance"
├─2116 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer ReqServer_ProcessManager"
├─2117 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master FileServerUpdate"
├─2118 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorkerQueue"
├─2119 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-0"
├─2121 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-1"
├─2128 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-2"
├─2129 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-3"
└─2130 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-4"
Apr 03 17:52:38 vm-202311071019-debian12 systemd[1]: Starting salt-master.service - The Salt Master Server...
Apr 03 17:53:44 vm-202311071019-debian12 systemd[1]: Started salt-master.service - The Salt Master Server.
Apr 03 18:02:11 vm-202311071019-debian12 salt-master[2111]: [ERROR ] Publish server binding pub to /var/run/salt/master/master_event_pub.ipc ssl=None
root@vm-202311071019-debian12:~# salt --versions-report
Salt Version:
Salt: 3007.0
Python Version:
Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]
Dependency Versions:
cffi: 1.16.0
cherrypy: unknown
dateutil: 2.8.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.3
libgit2: Not Installed
looseversion: 1.3.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.7
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 23.1
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.5.2
PyYAML: 6.0.1
PyZMQ: 25.1.2
relenv: 0.15.1
smmap: Not Installed
timelib: 0.3.0
Tornado: 6.3.3
ZMQ: 4.3.4
Salt Package Information:
Package Type: onedir
System Versions:
dist: debian 12.5 bookworm
locale: utf-8
machine: x86_64
release: 6.1.0-18-amd64
system: Linux
version: Debian GNU/Linux 12.5 bookworm
i've installed 3007.0 server and it gives me this error too, like here and like in #66282:
[ERROR ] Publish server binding pub to /run/salt/master/master_event_pub.ipc ssl=None
i see permissions in /run/salt/master/
3005.1:
srw-rw-rw- 1 salt salt 0 Apr 4 16:16 master_event_pub.ipc
srw------- 1 salt salt 0 Apr 4 16:16 master_event_pull.ipc
srw------- 1 salt salt 0 Apr 4 16:16 publish_pull.ipc
srwxr-xr-x 1 salt salt 0 Apr 4 16:16 workers.ipc
and with master 3007.0 i see
srw------- 1 salt salt 0 Apr 4 16:26 master_event_pub.ipc
srw------- 1 salt salt 0 Apr 4 16:26 master_event_pull.ipc
srw------- 1 salt salt 0 Apr 4 16:26 publish_pull.ipc
srwx------ 1 salt salt 0 Apr 4 16:26 workers.ipc
for now i have to roll back since it seems a 3007.0 issue
I was able to 'resolve' this by granting the salt user full access to all files under /var/run/salt/master. It may be overkill, but after running 'chmod 777 /var/run/salt/master/*' and restarting salt-master, it began working as designed.
@kossde You just gave every user full access to everything under /var/run/salt/master.
What you should have done is chown -R salt:salt /var/run/salt/master.
I recommend rebooting your master so /var/run/salt is recreated, then performing the above instead.
