Add Support for Conformal Prediction

[valeman]

Summary

This proposal requests the addition of Conformal Prediction methods to LightGBM for both regression and classification tasks. Conformal Prediction provides a layer of uncertainty quantification to predictions, which is currently missing in LightGBM.

https://github.com/valeman/awesome-conformal-prediction

Motivation

While LightGBM is a powerful and efficient library for gradient boosting, it lacks mechanisms for reliable modern uncertainty quantification, a feature present in some other machine learning libraries. The implementation of Conformal Prediction would enhance LightGBM's capabilities by providing statistically valid measures of uncertainty, which are crucial in many real-world scenarios where decision-making depends not just on predictions but also on the confidence in those predictions.

Description

The proposal includes the integration of two main types of Conformal Prediction:

Regression: Implementation of Inductive (split) and Conformalized Quantile Regression. This would allow users to obtain prediction intervals along with point predictions, offering a more complete picture of possible outcomes.

Classification: Implementation of the Venn-Abers Predictor. This approach extends the conformity measure to classification problems, providing valid probability estimates for each class.

These methods are aligned with LightGBM's focus on performance and scalability, and they would leverage LightGBM's existing infrastructure for regression and classification.

References

• MAPIE (Model-Agnostic Prediction Interval Estimator) implementation for Conformal Prediction: MAPIE Example
• Paper on Venn-Abers Predictors: NeurIPS 2015 Paper
• Talk by Vladimir Vovk on Conformal Prediction: YouTube Talk
• Tutorial on Conformal Prediction: COPA 2017 Tutorial
• Venn-Abers implementation by Ivan Petej: GitHub Repository
• Older implementation of Venn-Abers by Paolo Tocacceli: GitHub Repository
• Venn ABERS demo: GitHub Demo