salt
salt copied to clipboard
Published
20 hours ago •
saltstack
saltstack
[BUG] 3005.4 salt-minion dumps core when journald beacon is active on RHEL9.2 machine
Description The salt-minion service on RHEL 9.2 dumps core abruptly and produces the following core -
core
Feb 06 23:31:18 minion systemd-coredump[23973]: [🡕] Process 18182 (/usr/bin/python) of user 0 dumped core.
Stack trace of thread 18182:
#0 0x00007f4b97ad1dd2 real_journal_next (libsystemd.so.0 + 0x66dd2)
#1 0x00007f4b947b1de9 Reader_next (_reader.cpython-39-x86_64-linux-gnu.so + 0x5de9)
#2 0x00007f4b99525eb8 cfunction_call (libpython3.9.so.1.0 + 0x125eb8)
#3 0x00007f4b99518264 _PyObject_MakeTpCall (libpython3.9.so.1.0 + 0x118264)
#4 0x00007f4b9951519e _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x11519e)
#5 0x00007f4b9950eac5 _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10eac5)
#6 0x00007f4b9951c2c5 _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x11c2c5)
#7 0x00007f4b995101d4 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x1101d4)
#8 0x00007f4b9951c553 function_code_fastcall (libpython3.9.so.1.0 + 0x11c553)
#9 0x00007f4b995137d7 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x1137d7)
#10 0x00007f4b9950eac5 _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10eac5)
#11 0x00007f4b9951c2c5 _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x11c2c5)
#12 0x00007f4b99524a92 method_vectorcall (libpython3.9.so.1.0 + 0x124a92)
#13 0x00007f4b9958b8a1 _PyObject_VectorcallTstate.lto_priv.0 (libpython3.9.so.1.0 + 0x18b8a1)
#14 0x00007f4b994d2dba context_run (libpython3.9.so.1.0 + 0xd2dba)
#15 0x00007f4b9951787f cfunction_vectorcall_FASTCALL_KEYWORDS (libpython3.9.so.1.0 + 0x11787f)
#16 0x00007f4b99515d95 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x115d95)
#17 0x00007f4b9950eac5 _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10eac5)
#18 0x00007f4b9951c2c5 _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x11c2c5)
#19 0x00007f4b99524a92 method_vectorcall (libpython3.9.so.1.0 + 0x124a92)
#20 0x00007f4b995137d7 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x1137d7)
#21 0x00007f4b9950eac5 _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10eac5)
#22 0x00007f4b9951c2c5 _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x11c2c5)
#23 0x00007f4b99517cc7 _PyObject_FastCallDictTstate (libpython3.9.so.1.0 + 0x117cc7)
#24 0x00007f4b995234dc _PyObject_Call_Prepend (libpython3.9.so.1.0 + 0x1234dc)
#25 0x00007f4b995c2ca2 slot_tp_call (libpython3.9.so.1.0 + 0x1c2ca2)
#26 0x00007f4b99518264 _PyObject_MakeTpCall (libpython3.9.so.1.0 + 0x118264)
#27 0x00007f4b99514b20 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x114b20)
#28 0x00007f4b9951c553 function_code_fastcall (libpython3.9.so.1.0 + 0x11c553)
#29 0x00007f4b995101d4 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x1101d4)
#30 0x00007f4b9951c553 function_code_fastcall (libpython3.9.so.1.0 + 0x11c553)
#31 0x00007f4b995101d4 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x1101d4)
#32 0x00007f4b9950eac5 _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10eac5)
#33 0x00007f4b9951c2c5 _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x11c2c5)
#34 0x00007f4b99514b9e _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x114b9e)
#35 0x00007f4b9951c553 function_code_fastcall (libpython3.9.so.1.0 + 0x11c553)
#36 0x00007f4b99524a92 method_vectorcall (libpython3.9.so.1.0 + 0x124a92)
#37 0x00007f4b995137d7 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x1137d7)
#38 0x00007f4b9950eac5 _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10eac5)
#39 0x00007f4b9951c2c5 _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x11c2c5)
#40 0x00007f4b995177e6 _PyObject_VectorcallTstate.lto_priv.27 (libpython3.9.so.1.0 + 0x1177e6)
#41 0x00007f4b9950feff _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x10feff)
#42 0x00007f4b9951c553 function_code_fastcall (libpython3.9.so.1.0 + 0x11c553)
#43 0x00007f4b995101d4 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x1101d4)
#44 0x00007f4b9951c553 function_code_fastcall (libpython3.9.so.1.0 + 0x11c553)
#45 0x00007f4b995101d4 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x1101d4)
#46 0x00007f4b9951c553 function_code_fastcall (libpython3.9.so.1.0 + 0x11c553)
#47 0x00007f4b995101d4 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x1101d4)
#48 0x00007f4b9951c553 function_code_fastcall (libpython3.9.so.1.0 + 0x11c553)
#49 0x00007f4b995101d4 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x1101d4)
#50 0x00007f4b9950eac5 _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10eac5)
#51 0x00007f4b9951c2c5 _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x11c2c5)
#52 0x00007f4b995101d4 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x1101d4)
#53 0x00007f4b9950eac5 _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10eac5)
#54 0x00007f4b9951c2c5 _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x11c2c5)
#55 0x00007f4b995137d7 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x1137d7)
#56 0x00007f4b9951c553 function_code_fastcall (libpython3.9.so.1.0 + 0x11c553)
#57 0x00007f4b995101d4 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x1101d4)
#58 0x00007f4b9950eac5 _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10eac5)
#59 0x00007f4b9951c2c5 _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x11c2c5)
#60 0x00007f4b995249f1 method_vectorcall (libpython3.9.so.1.0 + 0x1249f1)
#61 0x00007f4b99510ebf _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x110ebf)
#62 0x00007f4b9951c553 function_code_fastcall (libpython3.9.so.1.0 + 0x11c553)
#63 0x00007f4b995101d4 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x1101d4)
Stack trace of thread 18359:
#0 0x00007f4b9914e83e epoll_wait (libc.so.6 + 0x14e83e)
#1 0x00007f4b975a1770 _ZN3zmq7epoll_t4loopEv (libzmq.so.5 + 0x2a770)
#2 0x00007f4b975d0421 thread_routine (libzmq.so.5 + 0x59421)
#3 0x00007f4b9909f802 start_thread (libc.so.6 + 0x9f802)
#4 0x00007f4b9903f450 __clone3 (libc.so.6 + 0x3f450)
Stack trace of thread 18350:
#0 0x00007f4b9914e83e epoll_wait (libc.so.6 + 0x14e83e)
#1 0x00007f4b975a1770 _ZN3zmq7epoll_t4loopEv (libzmq.so.5 + 0x2a770)
#2 0x00007f4b975d0421 thread_routine (libzmq.so.5 + 0x59421)
#3 0x00007f4b9909f802 start_thread (libc.so.6 + 0x9f802)
#4 0x00007f4b9903f450 __clone3 (libc.so.6 + 0x3f450)
Stack trace of thread 18360:
#0 0x00007f4b9914e83e epoll_wait (libc.so.6 + 0x14e83e)
#1 0x00007f4b975a1770 _ZN3zmq7epoll_t4loopEv (libzmq.so.5 + 0x2a770)
#2 0x00007f4b975d0421 thread_routine (libzmq.so.5 + 0x59421)
#3 0x00007f4b9909f802 start_thread (libc.so.6 + 0x9f802)
#4 0x00007f4b9903f450 __clone3 (libc.so.6 + 0x3f450)
Stack trace of thread 18351:
#0 0x00007f4b9914e83e epoll_wait (libc.so.6 + 0x14e83e)
#1 0x00007f4b975a1770 _ZN3zmq7epoll_t4loopEv (libzmq.so.5 + 0x2a770)
#2 0x00007f4b975d0421 thread_routine (libzmq.so.5 + 0x59421)
#3 0x00007f4b9909f802 start_thread (libc.so.6 + 0x9f802)
#4 0x00007f4b9903f450 __clone3 (libc.so.6 + 0x3f450)
Stack trace of thread 18183:
#0 0x00007f4b99144e4d __select (libc.so.6 + 0x144e4d)
#1 0x00007f4b995fa60d time_sleep (libpython3.9.so.1.0 + 0x1fa60d)
#2 0x00007f4b9951be17 cfunction_vectorcall_O (libpython3.9.so.1.0 + 0x11be17)
#3 0x00007f4b99514b9e _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x114b9e)
#4 0x00007f4b9950eac5 _PyEval_EvalCode (libpython3.9.so.1.0 + 0x10eac5)
#5 0x00007f4b9951c2c5 _PyFunction_Vectorcall (libpython3.9.so.1.0 + 0x11c2c5)
#6 0x00007f4b995137d7 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x1137d7)
#7 0x00007f4b9951c553 function_code_fastcall (libpython3.9.so.1.0 + 0x11c553)
#8 0x00007f4b995101d4 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x1101d4)
#9 0x00007f4b9951c553 function_code_fastcall (libpython3.9.so.1.0 + 0x11c553)
#10 0x00007f4b995101d4 _PyEval_EvalFrameDefault (libpython3.9.so.1.0 + 0x1101d4)
#11 0x00007f4b9951c553 function_code_fastcall (libpython3.9.so.1.0 + 0x11c553)
#12 0x00007f4b99524a92 method_vectorcall (libpython3.9.so.1.0 + 0x124a92)
#13 0x00007f4b995d6c5a t_bootstrap (libpython3.9.so.1.0 + 0x1d6c5a)
#14 0x00007f4b995d6c08 pythread_wrapper (libpython3.9.so.1.0 + 0x1d6c08)
#15 0x00007f4b9909f802 start_thread (libc.so.6 + 0x9f802)
#16 0x00007f4b9903f450 __clone3 (libc.so.6 + 0x3f450)
ELF object binary architecture: AMD x86-64
Setup The journald beacon is active on the machine as following -
journald beacon configuration
salt-call beacons.list
local:
beacons:
monitor-journald:
- enable: true
- services:
sshd:
PRIORITY: 6
SYSLOG_IDENTIFIER: sshd
su-5:
PRIORITY: 5
SYSLOG_IDENTIFIER: su
su-6:
PRIORITY: 6
SYSLOG_IDENTIFIER: su
su-7:
PRIORITY: 7
SYSLOG_IDENTIFIER: su
sudo:
PRIORITY: 5
SYSLOG_IDENTIFIER: sudo
- beacon_module: journald
- disable_during_state_run: true
Please be as specific as possible and give set-up details.
- [ ] on-prem machine
- [x] VM (KVM)
- [ ] VM running on a cloud service, please be explicit and add details
- [ ] container (Kubernetes, Docker, containerd, etc. please specify)
- [ ] or a combination, please be explicit
- [ ] jails if it is FreeBSD
- [x] classic packaging
- [ ] onedir packaging
- [x] used bootstrap to install
Steps to Reproduce the behavior Install salt-minion and activate journald beacon. Minion randomly dumps core.
Expected behavior The minion is expected to not dump core and stay in failed service state.
Screenshots If applicable, add screenshots to help explain your problem.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)Salt Version:
Salt: 3005.4
Dependency Versions:
cffi: 1.14.5
cherrypy: Not Installed
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.11.3
libgit2: Not Installed
M2Crypto: 0.38.0
Mako: Not Installed
msgpack: 1.0.3
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.20
pycrypto: Not Installed
pycryptodome: 3.14.0
pygit2: Not Installed
Python: 3.9.16 (main, Jan 4 2024, 00:00:00)
python-gnupg: Not Installed
PyYAML: 5.4.1
PyZMQ: 22.3.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: rhel 9.2 Plow
locale: utf-8
machine: x86_64
release: 5.14.0-284.11.1.el9_2.x86_64
system: Linux
version: Red Hat Enterprise Linux 9.2 Plow
Additional context Add any other context about the problem here.
