namespace problem with Building a Library on DBFS

[fissehab]

I created a repository of the R packages I need in DBFS using the steps outline here. Then when the cluster starts add the path for the packages as `## Add library to search path

 .libPaths(c("/dbfs/R_packages/", .libPaths()))`

The prophet package that I am using needs rlang package with version > 1.0.0. I have rlang with version 1.0.1 in /dbfs/R_packages/ but when I try to load prophet it fails saying that rlang is old. I tried to remove rlang from the namespace but it does not get removed; I get an error that htmltools has loaded it. How can I solve the issue?

library(prophet) ``Error in value[3L] : Package ‘rlang’ version 0.4.12 cannot be unloaded: Error in unloadNamespace(package) : namespace ‘rlang’ is imported by ‘htmltools’ so cannot be unloaded

Some( Error in value[3L]: Package ‘rlang’ version 0.4.12 cannot be unloaded: ) Error in value[3L]: Package ‘rlang’ version 0.4.12 cannot be unloaded: Error in unloadNamespace(package) : namespace ‘rlang’ is imported by ‘htmltools’ so cannot be unloaded``

[RafiKurlansik]

Thanks for reporting. Can you share which version of DBR you are on and the R packages you are trying to install?

[fissehab]

10.1 (includes Apache Spark 3.2.0, Scala 2.12) and the packages:

my_packages = c("prophet", "tidyverse","data.table","dplyr","fansi","utf8","stringr","fpp",
"quantmod","xts","lubridate","forecast","scales","readxl", "RPostgreSQL","RJDBC", "pracma", "tis", "xlsx","aws.iam", "tidyr", "readr", 
               "rJava", "RJDBC", 'tidymodels', "fma")

for (this_package in my_packages){
  install.packages(this_package, repos='http://cran.us.r-project.org', dependencies = TRUE)
}

[RafiKurlansik]

So after testing this out it doesn't appear to be an issue with adding the DBFS path to .libPaths(). Simply installing those packages and then trying to load prophet will result in the same issue. More investigation needed on why this is the case and if it is specific to this DBR or not.

[oaoutpost0]

If this is being done from within the Rstudio instance, it's possible that could be the reason the rlang package cannot be unloaded, since Rstudio depends on rlang and htmltools. https://docs.rstudio.com/ide/server-pro/latest/r_package_dependencies/r_package_dependencies.html