vowpal_wabbit
vowpal_wabbit copied to clipboard
VowpalWabbit
logging infrastructure for daemon mode
In production and staging environments, I find that it makes sense to output a timestamp along every error/warning message, for making troubleshooting effective. So long as the typical use case is serving in production through the daemon mode..
As I practically assume this is a patches welcome thing, any comments as to the feasibility of wiring that in, or special aspects to note in this, are welcome!
I would assume this means changing most uses of cout to a function call.
I agree it would be preferable for these logging messages to go though some sort of function call or even better a logger interface. That way the behavior of the logging can be changed such as dding a timestamp like you said or writing to a file
The essential issue here is that the error stream needs to be redirected into the report stream in daemon mode.
VW isn't setup to do this easily. The simplest clean solution is to catch a thrown exception, report that, and then kill the process (so all memory is cleaned up) relying on the master process to restart the daemon. This would be inefficient but memory safe.
VW now has more structured logging which can be redirected as required. It doesn't include timestamps but I am hesitant to add timestamps into VW. However, either a process consuming VW logs as a stream could inject a timestamp or you could use VW::initialize_experimental to have a custom log sink and determine timestamps there.