Metrics limited to a single Expected Calibration Error (ECE) formula

[thibaultcordier]

Describe the bug MAPIE offers only one way to calculate the ECE, but there are different estimators of the ECE in the literature (reference: https://arxiv.org/abs/2109.03480).

To Reproduce When using the expected_calibration_error(y_true: ArrayLike, y_scores: ArrayLike) function, the computed estimator of ECE is the class-specific ECE associated to class c (see https://arxiv.org/abs/2109.03480 in page 4).

To use this function correctly, the array y_true must be the boolean array indicating whether the example at index i is associated with the class considered (named c). So for any example at the index i, y_true[i] = 1 if this example is related to the considered class c.

Expected behavior It will be interesting to implement other ECE estimators, such as the confidence ECE presented in https://arxiv.org/abs/1706.04599.

Additional context Links for references:

• Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017, July). On calibration of modern neural networks. In International conference on machine learning (pp. 1321-1330). PMLR. (https://arxiv.org/abs/1706.04599)
• Posocco, N., & Bonnefoy, A. (2021). Estimating expected calibration errors. In Artificial Neural Networks and Machine Learning–ICANN 2021: 30th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 14–17, 2021, Proceedings, Part IV 30 (pp. 139-150). Springer International Publishing. (https://arxiv.org/abs/2109.03480)