projpred
projpred copied to clipboard
stan-dev
Computing LOOs in parallel
Since it is slow to compute LOOs it would be useful to be able to do it in parallel. Maybe through the future package.
This is possible in certain settings and we have a plan sketched out for the implementation. I hope we can get some development time soon and address this and other optimisation features. Thanks for the proposal. If you would like to participate in this we can have a chat about the necessary steps going forward.
Sure! I am currently trying to figure out how the for loop in loo_varsel could be rewritten to allow for parallelization. Which settings wouldn't allow for parallelization?
Sorry my comment was confusing. I meant that LOOs in parallel are indeed possible but in some cases you might want to place the parallelisation at different levels. We have a working branch where we are trying a different projpred workflow and one of the steps involves computing approximate LOO where the main bottleneck is for many posterior draws. In that case you want to do parallelisation for fitting posterior draws instead of LOO for several models.
To be clear, I initially meant in parallel in the posterior draw level. However, in this case I had 5 models and 6 CPU cores, so I made them parallel with future, which worked fine (after some initial failures, for some reason).
It is also possible to do LOOs in parallel in terms of data points, just to complement my previous answer, which, depending on the model and number of LOO evaluations, might make sense as well, particularly in problems where solving the individual projections is not extremely costly and the bottleneck is then to do many LOO evaluations.
Best, Alejandro On 22. Feb 2021, 18.45 +0200, Staffan Betnér [email protected], wrote:
To be clear, I initially meant in parallel in the posterior draw level. However, in this case I had 5 models and 6 CPU cores, so I made them parallel with future, which worked fine (after some initial failures, for some reason). — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.
In many cases approximate LOO is already a good estimate, and in that case we don't really care about number of datapoints as the matrix multiplication is reasonably fast, and the only bottleneck is the number of models. Nonetheless, both levels of parallelism are okay, it just depends on the particular problem which one returns more benefits. If you want to give a try at implementing a prototype I can help.
Sorry my comment was confusing. I meant that LOOs in parallel are indeed possible but in some cases you might want to place the parallelisation at different levels. We have a working branch where we are trying a different projpred workflow and one of the steps involves computing approximate LOO where the main bottleneck is for many posterior draws. In that case you want to do parallelisation for fitting posterior draws instead of LOO for several models.
+1 for parallelizing across (clusters of) posterior draws since I also think that the main bottleneck is usually the projection, not the CV, especially if the defaults for ndraws, nclusters, ndraws_pred, and nclusters_pred are as high as they are in the develop branch now.
In case of PSIS-LOO-CV, parallelization could then also be performed here: https://github.com/stan-dev/projpred/blob/0cabd73548eec32e9b675f15cd39d377a40f180a/R/cv_varsel.R#L366 and probably also here: https://github.com/stan-dev/projpred/blob/0cabd73548eec32e9b675f15cd39d377a40f180a/R/cv_varsel.R#L448
There is already a prototype working in workflow branch. If anyone is interested I can give some instructions. Basically, using varsel_cv in workflow branch as in cv_varsel in develop branch should probably work right away with a reference model or fit object.
Almost entirely resolved by PRs #235 and #422. The only remaining point is the cores = 1 thing from https://github.com/stan-dev/projpred/issues/77#issuecomment-801788960 but I'll add that to a projpred-specific to-do list that should go live soon. Closing this issue now.