Flask-MonitoringDashboard icon indicating copy to clipboard operation
Flask-MonitoringDashboard copied to clipboard

Results with very little evidence are also included in reports

Open dejagersh opened this issue 5 years ago • 0 comments

image

This example claims a 53% decrease in latency, but for these two intervals there are only two measurements for this endpoint. Obviously, ridiculous to include a result like this, given such little evidence.

I have two simple suggestions:

  • Include all results, but mention the size of the samples used. Add somewhere: "Based on two samples with 1 measurement and 2 measurements". This way people can make their own judgement about the validity of this result.
  • Exclude results that compared small samples -> Either sample needs to have at least 30 measurements for example.

These are two simple options that only take a few lines at most to implement. There are probably more sophisticated ways, but I'll research those during my thesis project :)

dejagersh avatar Apr 02 '20 16:04 dejagersh