Significancy threshold is too high for for mean latency reporting

[dejagersh]

Guys,

currently mean latency differences are only reported if the difference is at least 30%. Motivation was: differences less than 30% aren't really interesting. On second thought, I think even a change of a few percent can matter to some applications.

I suggest we lower this threshold or remove it all together.

What do you think?

[mircealungu]

First thought: use a reasonable default and let the individual user change it in the configuration?

Also, I thought 30% was also to ensure that we are not registering noise. Is this so? Do we have better statistics in the meantime?

[dejagersh]

Also, I thought 30% was also to ensure that we are not registering noise. Is this so?

Not necessarily. The P-value should be used for that. If it's just noise, the P-value will be greater than 0.05, meaning: not enough evidence supporting the hypothesis that there is a significant difference between the two samples.