Detect and warn if the samples are not normally distributed

[fitzgen]

trafficstars

From #138:

Ah, and one more thought: have we considered any statistical analysis that would look for multi-modal distributions (and warn, at least)? If we see that e.g. half of all runs of a benchmark run in 0.3s and half in 0.5s, and the distribution looks like the sum of two Gaussians, it may be better to warn the user "please check settings X, Y, Z; you seem to be alternating between two different configurations randomly" than to just present a mean of 0.4s with some wide variance, while the latter makes more sense if we just have a single Gaussian with truly random noise.

We can use the Shapiro-Wilk test to determine normality: https://en.wikipedia.org/wiki/Shapiro%E2%80%93Wilk_test

[fitzgen]

I don't really trust myself to implement Shapiro-Wilk myself, but I whipped up this script in R that checks the normality of count for each (arch, engine, wasm, phase, event) grouping:

#!/usr/bin/env Rscript

library(dplyr)

args = commandArgs(trailingOnly = TRUE)

if (length(args) != 1) {
    cat("Error: must supply path to sightglass CSV data\n")
    quit(status = 1)
}

data <- read.table(args[1], header = TRUE, sep = ",", stringsAsFactors = TRUE)

grouped <- data %>% group_by(arch, engine, wasm, phase, event)

error_count <- 0

add_error <- function (msg, key) {
    error_count <- error_count + 1
    cat(msg)
    cat(paste("    arch =", key$arch, "\n"))
    cat(paste("    engine =", key$engine, "\n"))
    cat(paste("    wasm =", key$wasm, "\n"))
    cat(paste("    phase =", key$phase, "\n"))
    cat(paste("    event =", key$event, "\n"))
    cat("\n")
}

### Normality

check_normality <- function (rows, key) {
    normality <- shapiro.test(rows$count)
    if (normality$p.value < 0.05) {
        add_error("Error: not normally distributed:\n", key)
    }
}

grouped %>% group_walk(check_normality)

quit(status = error_count)

So far, even when the benchmark results' distributions are definitely not bi-modal, they still aren't normal. Nor is the log of those results normal.

I think this technically invalidates the results of our t-test / effect size analyses, since these things assume that the data is normally distributed...