Details on how significance is calculated?

[LiFaytheGoblin]

Hi, sorry to be bothering you again!

I am currently looking at src/aequitas/bias.py to find out how the significance is calculated. Here is what I think I understood:

1. Check if sample group is normally distributed
2. a. If sample group is normally distributed, calculate whether sample group and ref. group have equal variances using levene b. If sample group is not normally distributed, calculate whether sample group and ref. group have equal variances using bartlett
3. a. if both groups have equal variances, perform independent 2 sample t-test b. if both groups have different variances, perform Welch's t-test

I have two questions: 1. Is this correct? I didn't understand what you mean with "sample" group at first. Now I understand that you mean the lists of binary encoded values that say for each entry of each group whether the entry belongs to whichever measure is relevant, fpr or fnr. It canalso be a list of scores, right?

2. Do we not check whether the ref. group is normally distributed? If not, why not? If we do, where? I found it! It is in an if condition https://github.com/dssg/aequitas/blob/a61ef33a55a8e21611425f13c5688bae6743f041/src/aequitas/bias.py#L521