Error: <callr_remote_error: Failed to move installed package at '/usr/local/lib/R/site-library/ps'>

[mskyttner]

Reproducible steps

Make a build with docker build . using the Dockerfile below:

FROM rocker/geospatial:4.0.3
RUN R -e 'install.packages("pak", repos = "https://r-lib.github.io/p/pak/dev/")'
RUN R -e 'pak::pkg_install("ps", ask = FALSE)'

Log

Stack trace:

12. (function (...) ...
13. base:::withCallingHandlers(cli_message = function(msg) { ...
14. get("pkg_install_do_plan", asNamespace("pak"))(...)
15. pkgdepends::install_package_plan(plan = plan, lib = lib, num_workers = num_ ...
16. base:::withCallingHandlers({ ...
17. pkgdepends:::handle_events(state, events)
18. pkgdepends:::handle_event(state, i)
19. proc$get_result()
20. processx:::process_get_result(self, private)
21. private$post_process()
22. pkgdepends:::install_extracted_binary(filename, lib_cache, pkg_cache, ...
23. base:::throw(new_fs_error("Failed to move installed package at {installed_p ...
24. base:::signalCondition(cond)
25. (function (e) ...
26. base:::stop(e)
27. (function (e) ...

Related issue

#175

[gaborcsardi]

For the record, #175 is not a related issue, that is a download error, this one is an installation error.

I am not sure why it happens, but it has to do something with the permissions of the library directory on that Docker image. E.g. this is successful:

> dir.create(lib <- tempfile())
> pak::pak("ps", lib = lib)

→ Will install 1 package.
→ Will download 1 CRAN package (115.13 kB).
+ ps   1.5.0 [bld][cmp][dl] (115.13 kB)
ℹ Getting 1 pkg (115.13 kB)
✔ Got ps 1.5.0 (source) (350.08 kB)
✔ Downloaded 1 package (350.08 kB) in 1.4s
ℹ Building ps 1.5.0
✔ Built ps 1.5.0 (478ms)
✔ Installed ps 1.5.0  (37ms)
✔ 1 + 0 pkgs | kept 0, updated 0, new 1 | downloaded 1 (350.08 kB) 2.6s

[gaborcsardi]

It is more clear when running the underlying pkgdepends commands:

> p <- pkgdepends::new_pkg_installation_proposal("ps")
> p$solve()
✔ Loading global cached package metadata ... done
> p$download()
ℹ Getting 1 pkg (115.13 kB)
✔ Cached copy of ps 1.5.0 (source) is the latest build
✔ No downloads needed, all packages are cached
> p$install()
ℹ Building ps 1.5.0
✔ Built ps 1.5.0 (441ms)
⸨▒▒▒▒▒▒▒▒▒▒▒▒▒⸩ | 📦  1/1     | ✅  0/1 ⠙ 1 | installing ps
Error: <install_filesystem_error in install_extracted_binary(filename, lib_cache, pkg_cache, lib,  ...:
 Failed to move installed package at '/usr/local/lib/R/site-library/ps'>

See `.Last.error.trace` for a stack trace.
Warning message:
In file.rename(installed_path, move_to) :
  cannot rename file '/usr/local/lib/R/site-library/ps' to '/usr/local/lib/R/site-library/_cache/ps/file3c256916315', reason 'Invalid cross-device link'

[mskyttner]

The permissions for that directory in the container:

drwxrwsr-x  1 root staff 4.0K Dec 13 19:33 site-library

Could it be a "staged install" thing where ps already is loaded and used by pak and some lock during a staged install phase prevents it from be "replaced"?

[gaborcsardi]

Could it be a "staged install" thing where ps already is loaded and used by pak and some lock during a staged install phase prevents it from be "replaced"?

No, locking is advisory on Linux.

It seems like it is not the permissions, but a Docker issue. We can try to work around it.

[gaborcsardi]

I am fairly sure that the issue is that the package (e.g. ps) is already installed on the parent image, and pak tries to move/rename its directory before the installation. For Docker this is a move/rename between devices. We can try to copy + delete if the move fails.

[ColinFay]

I also stumbled upon this issue when trying to update my R install scripts from {remotes} to {pak}.

Has there been any workaround for this?

So far I've been doing

packs <- as.data.frame(installed.packages())
if (!pak %in% packs$Package) {
	pak::pkg_install(
      pak
	)
  }

But that feels like a hack :)

[mjkanji]

@gaborcsardi Any updates on this or solutions that we can use to circumvent the problem? The above "hack" by ColinFay is only relevant for packages themselves and not their dependencies (e.g., if pkg depends on pkgY and pkgY is the one creating the issue, it's not particularly easy to conditionally exclude it using installed.packages() unless we look at the underlying solution from pkgdepends itself.

[mjkanji]

A (hopefully) more "resilient" version of the hack from @ColinFay that also excludes dependencies:

new_pkgs = c("tibble", "rlang", "fansi")
p <- pkgdepends::new_pkg_installation_proposal(new_pkgs)
p$solve()
sol <- p$get_solution()
sol_pkgs <- s$data$package
installed_pkgs <- installed.packages()[, "Package"]

# Remove any packages suggested by the solution which are already installed (regardless of version)
to_install <- setdiff(sol_pkgs, installed_pkgs )
pak::pkg_install(to_install, dependencies = FALSE)

An obvious disadvantage of this approach is that if one of the packages you have already installed is super old, pak might suggest updating it, but this approach overrides pak's suggestion and keeps the older version, which could potentially lead to its own set of problems...

[pat-s]

I also get biten by this in a docker setting.

The comment to try copy + delete was from the end of 2020. I would be interested to know if there have been some attempts to troubleshoot this or if it's possibly a dead end?

[schloerke]

My current work around...

Add a local library to your docker's Rprofile. Install your packages there.

# Use custom lib location
RUN mkdir -p '/custom/R/library' && echo ".libPaths('/custom/R/library')" >> `Rscript -e "cat(R.home())"`/etc/Rprofile.site

# Install pak
RUN R --quiet -e 'install.packages("pak", repos = sprintf("https://r-lib.github.io/p/pak/stable/%s/%s/%s", .Platform[["pkgType"]], R.Version()[["os"]], R.Version()[["arch"]]))'

# Install all pkgs
RUN R --quiet -e "pak::pkg_install('networkD3')"

By using a local lib:

• it doesn't match the parent R library, making it mutable
• packages within this lib be used first when running R

[telegott]

@schloerke I tried this workaround but I still get the error

Error: <callr_remote_error: Failed to move installed package at '/app/pak-library/classInt'>
 in process 30 
-->
<install_filesystem_error in install_extracted_binary(filename, lib_cache, pkg_cache, lib,  ...:
 Failed to move installed package at '/app/pak-library/classInt'>
 in process 

 Stack trace:

 12. (function (...)  ...
 13. base:::withCallingHandlers(cli_message = function(msg) { ...
 14. get("local_install_dev_deps_do_plan", asNamespace("pak"))(...)
 15. pak:::pkg_install_do_plan(proposal = NULL, lib = lib)
 16. pkgdepends::install_package_plan(plan = plan, lib = lib, num_workers = num_ ...
 17. base:::withCallingHandlers({ ...
 18. pkgdepends:::handle_events(state, events)
 19. pkgdepends:::handle_event(state, i)
 20. proc$get_result()
 21. processx:::process_get_result(self, private)
 22. private$post_process()
 23. pkgdepends:::install_extracted_binary(filename, lib_cache, pkg_cache,  ...
 24. base:::throw(new_fs_error("Failed to move installed package at {installed_p ...
 25. base:::signalCondition(cond)
 26. (function (e)  ...
 27. base:::stop(e)
 28. (function (e)  ...

 x Failed to move installed package at '/app/pak-library/classInt' 

The goal is to have a common base image, add test dependencies in a test layer and later deploy the base image with minimal changes as deploy so I don't have to rebuild the production libraries

Snippet from Dockerfile (.Rprofile is in the .dockerignore)

WORKDIR app
RUN mkdir -p pak-library
RUN echo ".libPaths('/app/pak-library')" > .Rprofile
COPY DESCRIPTION DESCRIPTION
COPY scripts/tool scripts/tool

RUN R -e "print(.libPaths())"
RUN set -xe \
    && Rscript -e 'install.packages("pak")' \
    && Rscript -e 'pak::local_install_deps()'
COPY . .

FROM base AS test
RUN Rscript -e "print(.libPaths())"
RUN set -xe \
    && Rscript -e 'pak::local_install_dev_deps()'

FROM base AS deploy
...

Do you have an idea what could be wrong here?

[gaborcsardi]

I guess the second pak call is trying to update a package that was installed in the first pak call.

[telegott]

@gaborcsardi this also happens with a minimal DESCRIPTION:

Imports: 
    sf
Suggests:
    lintr

and classInt does not seem to be a reverse dependency of lintr, but on the pak::install_local_dev_deps it indeed does try to update classInt, to the same version:

ℹ Loading metadata database
✔ Loading metadata database ... done

 
→ Will install 27 packages.
→ Will update 1 package.
→ Will download 28 CRAN packages (15.16 MB).
→ Will download 1 package with unknown size.
+ backports            1.4.1  [bld][cmp][dl] (26.20 kB)
+ callr                3.7.0  [bld][dl] (90.47 kB)
+ classInt     0.4-7 → 0.4-7  [bld][cmp][dl] (437.80 kB)
+ cli                  3.3.0  [bld][cmp][dl] (495.82 kB)
+ crayon               1.5.1  [bld][dl] (40.18 kB)
+ cyclocomp            1.1.0  [bld][dl] (8.41 kB)
+ desc                 1.4.1  [bld][dl] (80.67 kB)
+ digest               0.6.29 [bld][cmp][dl] (162.78 kB)
+ evaluate             0.15   [bld][dl] (25.62 kB)
+ glue                 1.6.2  [bld][cmp][dl] (106.51 kB)
+ highr                0.9    [bld][dl] (15.19 kB)
+ jsonlite             1.8.0  [bld][cmp][dl] (1.05 MB)
+ knitr                1.39   [bld][dl] (898.42 kB)
+ lazyeval             0.2.2  [bld][cmp][dl] (83.48 kB)
+ lintr                3.0.0  [bld][dl] (2.19 MB)
+ processx             3.6.1  [bld][cmp][dl] (161.75 kB)
+ ps                   1.7.1  [bld][cmp][dl] (128.66 kB)
+ R6                   2.5.1  [bld][dl] (63.42 kB)
+ remotes              2.4.2  [bld][dl] (152.54 kB)
+ rex                  1.2.1  [bld][dl] (93.58 kB)
+ rprojroot            2.0.3  [bld][dl] (59.94 kB)
+ stringi              1.7.6  [bld][cmp][dl] (8.03 MB)
+ stringr              1.4.0  [bld][dl] (135.78 kB)
+ withr                2.5.0  [bld][dl] (102.09 kB)
+ xfun                 0.31   [bld][cmp][dl] (126.06 kB)
+ xml2                 1.3.3  [bld][cmp][dl] (283.96 kB)
+ xmlparsedata         1.0.5  [bld][dl] (8.99 kB)
+ yaml                 2.3.5  [bld][cmp][dl] (94.53 kB)

I thought that workaround with a custom library path might fix this permission (?) issue as described. Is there another way to install only Suggests packages (of course there could be the same reverse dependency between a package in Imports and Suggests ...)

[gaborcsardi]

I am going to try to fix this now. A small reprex Dockerfile:

FROM rhub/r-minimal

WORKDIR /root

RUN installr -c -p
RUN Rscript -e 'pak::pkg_install("rlang")'

RUN Rscript -e 'pak::pkg_install("r-lib/rlang")'

[gaborcsardi]

OK, this is now fixed in pkgdepends, and the nightly devel Linux builds of pak. So this works now:

FROM rhub/r-minimal

WORKDIR /root

RUN installr -c

RUN Rscript -e 'install.packages("pak", repos = sprintf("https://r-lib.github.io/p/pak/devel/%s/%s/%s", .Platform$pkgType, R.Version()$os, R.Version()$arch))'

RUN Rscript -e 'pak::pkg_install("rlang")'

RUN Rscript -e 'pak::pkg_install("r-lib/rlang")'

[telegott]

@gaborcsardi awesome, thanks so much!