Investigate and possibly implement real-time tracing: qb-blackbox { --pid <PID> | --command <cmd> }
Currently, there seems to be an opportunity to get on-the-fly stream of messages temporarily hitting blackbox, enabler being on-disk materialized files.
This would also give an opportunity for a deepest look into the run possible, beside mere debugging/stracing/probing, which is rather cumbersome for not so isolated observations.
A similar capability is already available using signals: SIGUSR1 starts the collection of data, and SIGTRAP causes a write-out of the blackbox.
Thas misses several points:
- you talk just about pacemaker-specific provision
- blackbox has a limited capacity whereas stream is possibly "endless'
- may not solve infloop/deadlock problems when signal handling will never be served
Compared to big-gun solutions like attaching debugger, this would save one from active attendance on debugging session, stopping the process (making the timeouts kick in), expertise required from the user. This indeed assumes the trace messages are spread reasonably in the code.