sonic-buildimage
sonic-buildimage copied to clipboard
sonic-net
Close rsyslog plugin when rsyslog SIGTERM and EOF is sent to stream
Why I did it
Work item tracking
- Microsoft ADO (number only):27882794
How I did it
Add signalOnClose for omprog as well as close rsyslog plugin when receives an EOF.
How to verify it
UT (will add sonic-mgmt testcase for storming events with logs)
Which release branch to backport (provide reason below if selected)
- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
- [ ] 202111
- [ ] 202205
- [ ] 202211
- [ ] 202305
Tested branch (Please provide the tested image version)
- [ ]
- [ ]
Description for the changelog
Link to config_db schema for YANG module changes
A picture of a cute animal (not mandatory but encouraged)
@zbud-msft could you please check why the checker is skipped? would you please re-trigger it?
@lguohan This change is to address 2 issues
-
When rsyslogd is terminated, no signal is sent to child process of rsyslog_plugin meaning that rsyslog_plugin will be constantly trying to read from cin with no writer on the other end of the pipe. This leads to rsyslog_plugin process will constantly be reading via getline infinitely.
-
Because rsyslog is terminated and the spawned rsyslog_plugin is still alive, when rsyslog starts backup again, and log is triggered, a new rsyslog_plugin will be spawned for that rsyslog process, which can lead to many "ghost" rsyslog_plugin processes that will be at high CPU usage.
Cherry-pick PR to 202311: https://github.com/sonic-net/sonic-buildimage/pull/18968
Hi @zbud-msft, the change only handles SIG_TERM, but EOF of pipe is not handled yet. If rsyslog is killed by SIG_KILL or crash unexpectedly, the busy look is still there.
Discussed with Bing offline. From the code we are handling EOF. In loop we are checking getline(cin, line). If EOF is sent, getline(cin, line) will evaluate to false. UT also shows that we are replacing cin buffer with empty input stream, which will be treated as EOF when getline() is called by plugin->run.