Count Executions With Same Observer Value

[riesentoaster]

Is your feature request related to a problem? Please describe. I have a target that is unstable, i.e. for the same input I don't always get the same observation.

Describe the solution you'd like I would like to have a way to count how often a certain Observer observes the same value to get a frequency distribution and figure out how bad things are. This could be implemented in three levels:

1. A pure count of how often each observed value is measured — this would only really be useful with a single corpus entry and a NopMutator, but maybe a good start.
2. A count of how often each value is measured, along with an example input that triggers this execution
3. A count of how often each value is measured, along with a list of all inputs that triggered this execution — this would be really inefficient, but maybe useful for debugging.

Describe alternatives you've considered Right now, in a very hacky way, I just hash the observer value in the executor and append that hash to a file. It also gets put in some metadata in the corpus, so I can correlate them after the fact. This is wildly inefficient. And I'm using the single corpus entry/NopMutator strategy.

I've also attempted to implement this as a feedback that updates a count field in some metadata entry on each is_interesting call, but I could not get those updated metadata values to be written to disk. It's also a bit hacky, because in is_interesting, I only have the input, and not the testcase, so I have to manually extract that from state -> corpus -> testcase. If this were to work, it would basically provide solution 2. But I'm not sure it's possible, I don't know the internals well enough.

Lastly, one could write a feedback that keeps an internal representation of input-outcome values (e.g. HashMap with the Observer value as the key and a tuple of (count, example_input) as the value, to get solution level 2. These values could then be written to disk at a certain interval, similar to OnDiskTomlMonitor.

Additional context I'm just curious if anyone has an opinion on how this could be done in a pretty way.