Stratified(Balanced) Brier Score for Imbalanced datasets

[giorgosm3317]

Hello,

Forgive me if this request has already been made but I could not find it.

The idea is that the ordinary Brier Score in cases of imbalanced datasets may lead to miscalibration of the minority class as mentioned in this paper. The authors propose a stratified version of Brier score giving equal importance to all the classes. In the case of binary classification that would be the unweighted average of the brier scores of the majority and minority class.

It could also be extended to multiclass problems I believe.

Thanks!

[EmilHvitfeldt]

Hello @giorgosm3317 !

This is not a bad idea!