BiocManager
BiocManager copied to clipboard
Bioconductor
handle failed installations due to unwritable directory
A user trying to update a package that has been installed under an unwritable (e.g., windows Administrator) account is told that the package could not be updated.
This could be improved by telling the user that they do not have permission, or could be silently not reported (I think this is the approach taken in BiocInstaller).
update: @vjcitn suggests updating to a writeable location
@Jiefei-Wang mentioned this to me.
Same is true if it's called inside a container (i.e. singularity). Providing an additional writeable directory (.libPaths("lib")), doesn't eliminate the problem. Ideally, BiocManager would install the updated packages in the writeable folder and respect precedence.
If explicitly installed with BiocManager::install(pkgs) it gets installed in the additional writable folder. However BiocManager::install() still complains that it cannot be updated (although it's twice their once in the none writable and once in the writable). Further, BiocManager::validate() complains about outdated packages in the none-writable folder although the new versions are installed in the additional dir.
Thx!
This seems to be a windows issue and/or old.packages feature.
When testing this on Linux (see gist here), old.packages() does not recognize any out-of-date packages in unwriteable directories (mode = "0400"). It just ignores that location in .libPaths().
Therefore, our filter in https://github.com/Bioconductor/BiocManager/blob/master/R/install.R#L177 is not
even triggered because old.packages will not show / register old packages in unwriteable directories (line 172).
I think, in the gist, you want to set the unwritedir to the first rather than last lib path.
I've done that and ran everything again and I get the same effect. The package only shows out-of-date when I change the directory mode to "0700". See the updated gist
ok next iteration -- 0400 is too restrictive, the directory can't be 'exec'd so e.g., it's content isn't readable. Try 0500 (read + exec).
In the Linux container (singularity) all users (except root) are allowed to read + exec. Anyway container content cannot be changed during runtime. Only changes in external path is possible.
Container: https://www.singularity-hub.org/collections/2069 Container definition: https://github.com/ii-bioinfo/R3.5.1_Bioc3.8_rstudio1.1.463/blob/master/Singularity
Run R from container:
singularity run --app R shub://ii-bioinfo/R3.5.1_Bioc3.8_rstudio1.1.463
Update BiocManager to writable dir:
dir.create("~/libs")
.libPaths("~/libs")
BiocManager::install("BiocManager")
Bioconductor version 3.8 (BiocManager 1.30.3), R 3.5.1 (2018-07-02)
Installing package(s) 'BiocManager'
trying URL 'https://cran.rstudio.com/src/contrib/BiocManager_1.30.4.tar.gz'
Content type 'application/x-gzip' length 242304 bytes (236 KB)
==================================================
downloaded 236 KB
* installing *source* package ‘BiocManager’ ...
** package ‘BiocManager’ successfully unpacked and MD5 sums checked
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded
* DONE (BiocManager)
The downloaded source packages are in
‘/tmp/RtmptJvjYW/downloaded_packages’
installation path not writeable, unable to update packages: AnnotationHub,
backports, BiocManager, BiocParallel, Biostrings, biovizBase, bookdown,
ensembldb, GenomeInfoDb, GenomicAlignments, Gviz, htmlTable, httpuv, httr,
jsonlite, knitr, littler, markdown, openssl, pillar, RBGL, Rcpp, rmarkdown,
rtracklayer, S4Vectors, tibble, VariantAnnotation, class, codetools, mgcv,
survival
Reload newest version and try to validate:
library(BiocManager)
BiocManager::valid()$out_of_date["BiocManager",]
Complains about old installed version:
Package LibPath
"BiocManager" "/usr/local/lib/R/site-library"
Installed Built
"1.30.3" "3.5.1"
ReposVer Repository
"1.30.4" "https://cran.rstudio.com/src/contrib"
Warning message:
27 packages out-of-date; 0 packages too new
Confirm that session has new version although valid complains:
sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux 9 (stretch)
Matrix products: default
BLAS: /usr/lib/openblas-base/libblas.so.3
LAPACK: /usr/lib/libopenblasp-r0.2.19.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=C
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] BiocManager_1.30.4
loaded via a namespace (and not attached):
[1] compiler_3.5.1 tools_3.5.1
Permissions of BiocManager in container:
singularity shell --app R shub://ii-bioinfo/R3.5.1_Bioc3.8_rstudio1.1.463
ls -l /usr/local/lib/R/site-library
drwxrwxr-x 7 root root 132 Nov 6 17:18 BiocManager
I am not sure it is correctly using the first location.
> BiocManager::install("DESeq2")
Bioconductor version 3.9 (BiocManager 1.30.4), R 3.6.1 (2019-07-05)
Installing package(s) 'DESeq2'
trying URL 'https://bioconductor.org/packages/3.9/bioc/src/contrib/DESeq2_1.24.0.tar.gz'
Content type 'application/x-gzip' length 2056901 bytes (2.0 MB)
==================================================
downloaded 2.0 MB
* installing *source* package ‘DESeq2’ ...
** using staged installation
** libs
g++ -std=gnu++11 -I"/usr/local/lib/R/include" -DNDEBUG -I"/home/rsettlag/R/OOD/3.6.1/Rcpp/include" -I"/home/rsettlag/R/OOD/3.6.1/RcppArmadillo/include" -I/usr/local/include -fpic -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c DESeq2.cpp -o DESeq2.o
g++ -std=gnu++11 -I"/usr/local/lib/R/include" -DNDEBUG -I"/home/rsettlag/R/OOD/3.6.1/Rcpp/include" -I"/home/rsettlag/R/OOD/3.6.1/RcppArmadillo/include" -I/usr/local/include -fpic -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c RcppExports.cpp -o RcppExports.o
g++ -std=gnu++11 -shared -L/usr/local/lib/R/lib -L/usr/local/lib -o DESeq2.so DESeq2.o RcppExports.o -L/usr/local/lib/R/lib -lRlapack -lopenblas -lgfortran -lm -lquadmath -L/usr/local/lib/R/lib -lR
installing to /home/rsettlag/R/OOD/3.6.1/00LOCK-DESeq2/00new/DESeq2/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (DESeq2)
The downloaded source packages are in
‘/tmp/RtmpSeu0lO/downloaded_packages’
installation path not writeable, unable to update packages: bookdown, rgl, rmarkdown, rticles, sys, xfun, boot, foreign,
nlme
Looks to me as it has the correct path above, per
> .libPaths()
[1] "/home/rsettlag/R/OOD/3.6.1" "/usr/local/lib/R/site-library" "/usr/local/lib/R/library"
But, in the end, the above failed, and when I do the below, it works:
> BiocManager::install("DESeq2",lib=.libPaths()[1])
Bioconductor version 3.9 (BiocManager 1.30.4), R 3.6.1 (2019-07-05)
Installing package(s) 'DESeq2'
trying URL 'https://bioconductor.org/packages/3.9/bioc/src/contrib/DESeq2_1.24.0.tar.gz'
Content type 'application/x-gzip' length 2056901 bytes (2.0 MB)
==================================================
downloaded 2.0 MB
* installing *source* package ‘DESeq2’ ...
** using staged installation
** libs
g++ -std=gnu++11 -I"/usr/local/lib/R/include" -DNDEBUG -I"/home/rsettlag/R/OOD/3.6.1/Rcpp/include" -I"/home/rsettlag/R/OOD/3.6.1/RcppArmadillo/include" -I/usr/local/include -fpic -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c DESeq2.cpp -o DESeq2.o
g++ -std=gnu++11 -I"/usr/local/lib/R/include" -DNDEBUG -I"/home/rsettlag/R/OOD/3.6.1/Rcpp/include" -I"/home/rsettlag/R/OOD/3.6.1/RcppArmadillo/include" -I/usr/local/include -fpic -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g -c RcppExports.cpp -o RcppExports.o
g++ -std=gnu++11 -shared -L/usr/local/lib/R/lib -L/usr/local/lib -o DESeq2.so DESeq2.o RcppExports.o -L/usr/local/lib/R/lib -lRlapack -lopenblas -lgfortran -lm -lquadmath -L/usr/local/lib/R/lib -lR
installing to /home/rsettlag/R/OOD/3.6.1/00LOCK-DESeq2/00new/DESeq2/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (DESeq2)
The downloaded source packages are in
‘/tmp/RtmpSeu0lO/downloaded_packages’
All of this was in a Singularity container based on the Docker rocker/rstudio image.
In my experience (not sure if while installing Rgraphviz, or BiocManager altogether) boot, foreign and nlme were the packages ending up being written into the R install folder (i.e. not working without administrator permissions)
Putting aside I really don't know who would be pulling them in, what's so special about them?
These are 'recommended' packages, almost certainly installed when R was installed
> pkgs = installed.packages()
> as_tibble(pkgs) %>% filter(Priority %in% "recommended") %>% pull(Package)
[1] "mgcv" "boot" "class" "cluster" "codetools"
[6] "foreign" "KernSmooth" "lattice" "MASS" "Matrix"
[11] "mgcv" "nlme" "nnet" "rpart" "spatial"
[16] "survival"
Hi Bob, @rsettlage
I don't see the difference between the two calls.
lib is set to .libPaths()[1] internally when inspecting the code. See
BiocManager::.install arguments.
It may be possible that the singularity container configuration could be changing that.
@mirh These packages come with the R distribution and should be located in the r-release/R/lib/R/library. BiocManager doesn't touch those packages. They could show up in .libPaths()[1] if you install them explicitly à la BiocManager::install("boot"), for example.
@LiNk-NY I don't think Singularity is doing anything special in this case. I am Singularity pulling the Docker image and then running that. The only thing I am doing special in Singularity is setting /tmp to a directory outside the container users have read/write permissions to. It looks to me as if the error is more related to the desire to update packages that are installed in unwritable directories. I am guessing that when I give it a single path, it force updates those packages to the given path vs trying to update them in place.
These are 'recommended' packages, almost certainly installed when R was installed
Nope. I have the same R 3.6.1 for windows installed on my desktop (where I never tried to install stuff in these neighborhoods) and laptop (where I had to install rgraphviz). And only on the latter I have those packages. With their folders dated exactly with the time I gave the install command, rather than when I installed the whole enviroment to begin with.
I didn't touch any default value/setting whatsoever btw.
Getting same unwriteable error outside of singularity on a centos 7.7 kernel, Bioconductor version 3.10 (BiocManager 1.30.10), R 3.6.2 (2019-12-12)
do not have admin privileges. Set R_LIBS_USER prior to installing packages.
using lib=.libPaths()[1] like @rsettlage solved the issue
We are experiencing the same issue where the .libPaths()[1] is writable but the shared library paths are not (EasyBuild installation, not in any container).
Vanilla install.packages works as usual.
The workaround with libPaths()[1] also works for us.
BiocManager::install("AnnotationDbi",lib=.libPaths()[1])
However, BiocManager::install() will still list AnnotationDbi and not being able to update. Seems the .libPaths() order is not properly respected here.
> BiocManager::install(lib=.libPaths()[1])
Bioconductor version 3.11 (BiocManager 1.30.10), R 4.0.0 (2020-04-24)
> BiocManager::install()
Bioconductor version 3.11 (BiocManager 1.30.10), R 4.0.0 (2020-04-24)
Installation path not writeable, unable to update packages: AnnotationDbi, AnnotationHub, biglm,...
What is super odd in my case, if I do this via Rstudio, it all works correctly. I can do .libPaths() and the order is as expected, user first, shared/system second and third. If I instead get into R via the command line and look at the library paths, they are reversed.
BiocManager is checking all packages to see if they require update. With .libPaths()[1] provided explicitly, 'all' packages are those in the first (typically user) directory. With the implicit .libPaths(), it is all paths including system locations. Someone has installed AnnotationDbi etc in one of these paths, and with permissions (e.g., as administrator) that do not allow them to be updated. This is working as expected, and differs from install.packages() because install.packages() does not check whether packages need to be updated. There is an argument BiocManager::install(update = FALSE) that will stop the update check (but then the integrity of the installation is not validated or maintained).
@rsettlage this sounds like a configuration problem with your system, maybe a command line alias to R? rather than an issue on BiocManager?
@mtmorgan from a user point of view, I would expect the update check to see if the package that would be loaded by e.g. library("AnnotationDbi") is outdated or not. Or maybe report there are multiple versions installed. The current reporting is a bit confusing, especially in combination with the error about not writable installation paths - even though there is a user library path specified that is writable.
I'm having the same problem as scholtalbers when trying to update to 3.12, with the added bonus of having no control over .libPaths()[2], where the packages were originally installed by our sysad (not writtable by me, and also cannot be removed; if I do .libPaths('') my local library disappears, but the other one is still there).
I can update everything using the output from BiocManager::valid(), which installs the current version into .libPaths()[1], and which then get loaded preferentially. But I still get the out of date message for all packages every time I install something, and have no granular information over what actually is and isn't out of date. It would be nice to find a way to solve this!
Hi Irene, @ireneg
If you're updating to 3.12 from an older version, we recommend creating a new user write-able folder and
setting an R_LIBS_USER variable.
See the Managing multiple versions section in the BiocManager vignette.
As Martin said previously, BiocManager checks across all folders in .libPaths() for out-of-date packages even
if they are in an unwriteable directory. We are working on a filter (.package_filter_unwriteable)
for excluding these packages since the user is unable to modify them. Try using BiocManager::install("Bioconductor/BiocManager") until we can get a new package version uploaded.
PS. Please tag me @LiNk-NY so that I can get a notification.
@LiNk-NY the point shouldn't be that they are unwriteable per se (it's perfectly normal to have at least one such folder, and I would indeed expect an error when a package cannot be installed), but that you shouldn't try to write to the system/admin folder.
I mean, why can't you just use install.packages, or the getWritablePathname logic they employ behind the scenes?
@mirh
We can't control what the user has in their .libPaths but we do control where to write packages to. Any old
packages that need updating in an unwriteable location are ignored in .package_filter_unwriteable AFAICT.
We have a layer above install.packages to check for old CRAN and Bioconductor packages that need
updating, we only want to make sure that we pass along the packages that need to go into install.packages
this includes excluding those that are in unwriteable locations.