Metric value for orleans-grains can sometimes be negative

[randrewy]

Someties orleans-grains(microsoft_orleans_orleans_grains) metric repot negative values in our environment. I was not able to find anything in common for those types of grains i've seen, so presumably it can happen with any grain type (I've seen both ordinary and stateless workers)

We are using Orleans 8.0.0 with multiple silos, but it doesn't matter. The same happens on single silo from time to time.

Here is a screen from our prod env for last 6 hours (some graphs represent same GrainType but on different silos):

And this screen was reported in Discord:

And it can happen on system grains (this is single silo run):