reactor_cpu_steal_time_ms returns negative value

[amnonh]

scylla_reactor_cpu_steal_time_ms

sm::make_counter("cpu_steal_time_ms", [this] () -> int64_t { return total_steal_time() / 1ms; },
./seastar/src/core/reactor.cc

Returns a negative value

# TYPE scylla_reactor_cpu_steal_time_ms counter
scylla_reactor_cpu_steal_time_ms{shard="0"} -442
scylla_reactor_cpu_steal_time_ms{shard="1"} -604
scylla_reactor_cpu_steal_time_ms{shard="2"} -627
scylla_reactor_cpu_steal_time_ms{shard="3"} -622

[travisdowns]

This happens because steal time is "awake time (wall clock)" minus "cpu thread time (get_rusage style CPU time)", but the awake time can be under-counted because sleep time is over-counted (as it's not possible to determine exactly the true sleep time, only get timestamps before and after a period you think might involve a sleep).

The result is that when there is little to no true steal, CPU time will exceed measured awake wall clock time, resulting in negative steal for that interval and if that's frequent enough the absolute value of the counter can be negative too.