The rules need to define how consistent runs need to be

[wvaske]

One of the metrics reported with results submissions is the standard deviation of the primary metric.

The rules currently imply a requirement for consistency of runs but we don't have them defined. We should have a defined rules as to the allowed standard deviation.