iml icon indicating copy to clipboard operation
iml copied to clipboard

iml complains about future.globals.maxSize even when no future plan is set

Open jeffkeller-einc opened this issue 9 months ago • 0 comments

It seems as though {iml} always uses future.apply functions even when no future::plan() has been set, as you can see here:

library(iml)
library(randomForest)

data("Boston", package = "MASS")

rf <- randomForest(medv ~ ., data = Boston, n.trees = 10)
X <- Boston[which(names(Boston) != "medv")]
predictor <- Predictor$new(rf, data = X, y = Boston$medv)


options(future.globals.maxSize = 1)

FeatureImp$new(predictor, loss = "mape")


# Error in `getGlobalsAndPackages()`:
# ! The total size of the 2 globals exported is 56.59 MiB.. This exceeds the maximum allowed size of 1 bytes (option 'future.globals.maxSize'). There are two globals: ‘FUN’ (28.30 MiB of class ‘function’) and ‘MoreArgs’ (28.29 MiB of class ‘list’)

I discovered this after retraining a model whose training data had grown, pushing it just above the default future.globals.maxSize=500 MB. I was really confused at first because I explicitly chose not to configure a future::plan because I have a predict.function that handles parallelization.

I think {iml} should have a more explicit way of enabling parallelization to avoid confusion. It would be a better developer experience to not need to read the parallelization vignette in order to debug the default single-threaded behavior of the package.

jeffkeller-einc avatar Mar 25 '25 16:03 jeffkeller-einc