Enh: threaded/non-blocking/deferred trace writing

[rbtcollins]

Event writing is currently blocking - https://github.com/andjo403/rs_tracing/blob/master/src/lib.rs#L287 - which has two effects - it adds in-process latency and it will also add IO contention when the process is a particularly IO heavy one. It would be good to permit that to be managed (e.g. shove it off to a writer thread thats rate limited / use a mem queue or other techniques to avoid distorting the thing being measured.

[andjo403]

my hope was to not need to do this, do not know how mush difference it makes when I use the BufWriter (default 8kb write buffer) maybe can make it possible to make a bigger buffer.

[andjo403]

or you are talking about if two threads try to trace at the same time one thread is blocking the other

[rbtcollins]

I need to do some finer measurements I think, but yeah thats the concern I have. I'm measuring things doing disk ops and pushing up latencies on IO way up from time to time, and the trace work adds in both more work for the disk (not much true though that depends on how much you trace), but also can potentially stall the writing thread for substantial untraced periods and so on. Having the tracing load as explicitly as far out of the workload path as possible would reduce 'where is the time' going concerns is all :). Its a wishlish item - at this point, short of the finer measurements, I cannot say that its a pressing concern.

If IO was backed up the the mutex today would indeed cause blocking - special case of the general concern.