watchdogd

watchdogd copied to clipboard

Hi

For detecting thrashing I think it could be useful to monitor the memory pressure.

some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0

The “some” line indicates the share of time in which at least some tasks are stalled on a given resource.

The “full” line indicates the share of time in which all non-idle tasks are stalled on a given resource simultaneously. In this state actual CPU cycles are going to waste, and a workload that spends extended time in this state is considered to be thrashing. This has severe impact on performance, and it’s useful to distinguish this situation from a state where some tasks are stalled but the CPU is still doing productive work. As such, time spent in this subset of the stall state is tracked separately and exported in the “full” averages.

https://www.kernel.org/doc/html/latest/accounting/psi.html

What do you think? We have swap enabled on all our systems, which is part of the reason I'm asking for this.

Sure, memory pressure is a very interesting metric! I have no time to work on this myself, even though it's not hard writing a plugin for watchdogd. Unless someone beats be to it, I'll have a look once I circle back to this project (we just did a release).

What about tracking hangs or power cuts? Should be trivial to design a mechanism to write every minute on one or more dedicated log files, trimming them at periodic intervals.

Sure, anything's possible. Like I said, though, no time to implement requests. Pull requests are welcome :sunglasses::+1: