linfa icon indicating copy to clipboard operation
linfa copied to clipboard

Add boosting support to linfa-ensemble

Open yfnaji opened this issue 2 months ago • 2 comments

I was looking into the linfa-ensemble crate and noticed that it currently does not include a boosting mechanism. The current ensemble implementation appears to be quite generalised, so we could either extend EnsembleLearner to support boosting methods or (preferably) create a new struct dedicated to boosting, while keeping EnsembleLearner as the dedicated bagging struct (and rename EnsembleLearner -> Bagging).

I would like your thoughts on this - I’d be happy to work on this and write an initial implementation.

yfnaji avatar Oct 19 '25 11:10 yfnaji

In my mind EnsembleLearner is indeed the most generalized and more specialized algorithms composes with it.
You might be interested in discussing the implementation in #410 to target a smooth integration of your boosting method implementation.

relf avatar Oct 19 '25 15:10 relf

A gradient boosting machine (using decision trees) effectively requires regression trees since the algorithm operates on continuous residuals rather than discrete class labels. It appears that this package does not yet support regression trees as mentioned in the documentation and this issue.

I can work on adding regression tree support via a separate PR. Once that’s done, we can proceed with implementing gradient boosting. To keep things manageable, I will focus on implementing gradient boosting specifically for decision trees first, and then generalise to other base learners later.

yfnaji avatar Oct 24 '25 22:10 yfnaji