Adding a package that requires a newer version of a cache package causes build to fail

[zkamvar]

Problem

This is an example:

If you have a package cache that contains the {rmarkdown} package version 2.11 and add {bookdown} (which tracks the CRAN version of {rmarkdown} closely), you end up with a problem in the package discovery stage where the cache can not be hydrated because the dependency is not the right version.

Why would you add bookdown to a lesson? If you were migrating your content from a bookdown project, you might end up with a yaml header that looks like:

output: 
  bookdown::html_document2: 
    keep_

{renv} would see this and say "hey! you need bookdown!" and then this happens:

Log of output from GitHub Actions

 ℹ Consent to use package cache provided
  * "~/.local/share/renv/" has been created.
  → Searching for and installing available dependencies
  Finding R package dependencies ... Done!
  * Discovering package dependencies ... Done!
  * Copying packages into the cache ... Done!
  * Resolving missing dependencies  ... 
  Retrieving 'https://cran.rstudio.com/src/contrib/bookdown_0.26.tar.gz' ...
  	OK [downloaded 830.9 Kb in 0.3 secs]
  Installing bookdown [0.26] ...
  	OK [built from source]
  Moving bookdown [0.26] into the cache ...
  	OK [moved to cache in 0.31 milliseconds]
  → Restoring any dependency versions
  The following package(s) will be updated:
  
  # CRAN ===============================
  - evaluate    [repo: RSPM -> CRAN; ver: 0.15 -> 0.14]
  - glue        [repo: RSPM -> CRAN; ver: 1.6.2 -> 1.6.1]
  - jsonlite    [repo: RSPM -> CRAN; ver: 1.8.0 -> 1.7.3]
  - knitr       [repo: RSPM -> CRAN; ver: 1.39 -> 1.37]
  
   Process 10599:
   1. sandpaper::manage_deps(path = wd, quiet = FALSE)
   2. callr::r(func = callr_manage_deps, args = args, show = !quiet,  ...
   3. callr:::get_result(output = out, options)
   4. throw(newerr, parent = remerr[[2]])
  
   x callr subprocess failed: aborting snapshot due to pre-flight validation failure 
  
   Process 10655:
   16. (function (path, repos, snapshot, lockfile_exists)  ...
   17. renv::snapshot(project = path, lockfile = renv_lock, prompt = FALSE)
   18. base:::stop("aborting snapshot due to pre-flight validation failure")
   19. base:::.handleSimpleError(function (e)  ...
   20. h(simpleError(msg, call))
  
   x aborting snapshot due to pre-flight validation failure 
  
  Execution halted
  Error: Process completed with exit code 1.

Reprex

tmp <- tempfile()
dir.create(tmp)
gert::git_clone("https://github.com/zkamvar/Basic_Python", path = tmp, branch = "error-example")
sandpaper::manage_deps(tmp)
#> ℹ Consent to use package cache provided
#> → Searching for and installing available dependencies
#> Finding R package dependencies ... Done!
#> * Discovering package dependencies ... Done!
#> * Copying packages into the cache ... Done!
#> → Restoring any dependency versions
#> The following package(s) will be updated:
#> 
#> # CRAN ===============================
#> - evaluate    [0.15 -> 0.14]
#> - glue        [1.6.2 -> 1.6.1]
#> - jsonlite    [1.8.0 -> 1.7.3]
#> - knitr       [repo: RSPM -> CRAN; ver: 1.39 -> 1.37]
#> - magrittr    [2.0.3 -> 2.0.2]
#> - rlang       [1.0.2 -> 1.0.0]
#> - rmarkdown   [repo: RSPM -> CRAN; ver: 2.14 -> 2.11]
#> - tinytex     [0.38 -> 0.36]
#> - xfun        [0.30 -> 0.29]
#> - yaml        [2.3.5 -> 2.2.2]
#> - renv        [* -> 0.15.2]
#> 
#> Installing evaluate [0.14] ...
#>  OK [linked cache]
#> Installing glue [1.6.1] ...
#>  OK [linked cache]
#> Installing jsonlite [1.7.3] ...
#>  OK [linked cache]
#> Installing xfun [0.29] ...
#>  OK [linked cache]
#> Installing magrittr [2.0.2] ...
#>  OK [linked cache]
#> Installing yaml [2.2.2] ...
#>  OK [linked cache]
#> Installing knitr [1.37] ...
#>  OK [linked cache]
#> Installing rlang [1.0.0] ...
#>  OK [linked cache]
#> Installing tinytex [0.36] ...
#>  OK [linked cache]
#> Installing rmarkdown [2.11] ...
#>  OK [linked cache]
#> Installing renv [0.15.2] ...
#>  OK [linked cache]
#> The following package(s) have been updated:
#> 
#>  glue [installed version 1.6.1  != loaded version 1.6.2 ]
#>  yaml [installed version 2.2.2  != loaded version 2.3.5 ]
#>  renv [installed version 0.15.2 != loaded version 0.15.4]
#> 
#> Consider restarting the R session and loading the newly-installed packages.
#> 
#> → Recording changes in lockfile
#> The following package(s) have unsatisfied dependencies:
#> 
#>  bookdown requires rmarkdown (>= 2.13), but version 2.11 is installed
#> 
#> Consider updating the required dependencies as appropriate.
#> 
#> Warning message:
#> This project is configured to use R version '4.1.2', but '4.2.0' is currently being used.
#> Error: callr subprocess failed: aborting snapshot due to pre-flight validation failure

^{Created on 2022-05-09 by the reprex package (v2.0.1)}

Proposed Solution: update_cache() followed by manage_deps()

This did not work because there was a pre-flight validation failure for the snapshot.

Proposed fix via `update_cache()`

sandpaper::update_cache(tmp, prompt = FALSE)
#> Warning: This project is configured to use R version '4.1.2', but '4.2.0' is
#> currently being used.
#> * Querying repositories for available source packages ... Done!
#> * Checking for updated packages ... Done!
#> Installing evaluate [0.15] ...
#>  OK [linked cache]
#> Installing glue [1.6.2] ...
#>  OK [linked cache]
#> Installing jsonlite [1.8.0] ...
#>  OK [linked cache]
#> Installing magrittr [2.0.3] ...
#>  OK [linked cache]
#> Installing yaml [2.3.5] ...
#>  OK [linked cache]
#> Installing knitr [1.39] ...
#>  OK [linked cache]
#> Installing renv [0.15.4] ...
#>  OK [linked cache]
#> Installing rlang [1.0.2] ...
#>  OK [linked cache]
#> Installing tinytex [0.38] ...
#>  OK [linked cache]
#> Installing rmarkdown [2.14] ...
#>  OK [linked cache]
#> Error in renv::snapshot(lockfile = renv::paths$lockfile(project = path), : aborting snapshot due to pre-flight validation failure
sandpaper::manage_deps(tmp)
#> ℹ Consent to use package cache provided
#> → Searching for and installing available dependencies
#> Finding R package dependencies ... Done!
#> → Restoring any dependency versions
#> The following package(s) will be updated:
#> 
#> # CRAN ===============================
#> - evaluate    [0.15 -> 0.14]
#> - glue        [1.6.2 -> 1.6.1]
#> - jsonlite    [1.8.0 -> 1.7.3]
#> - knitr       [repo: RSPM -> CRAN; ver: 1.39 -> 1.37]
#> - magrittr    [2.0.3 -> 2.0.2]
#> - renv        [repo: RSPM -> CRAN; ver: 0.15.4 -> 0.15.2]
#> - rlang       [1.0.2 -> 1.0.0]
#> - rmarkdown   [repo: RSPM -> CRAN; ver: 2.14 -> 2.11]
#> - tinytex     [0.38 -> 0.36]
#> - yaml        [2.3.5 -> 2.2.2]
#> 
#> Installing evaluate [0.14] ...
#>  OK [linked cache]
#> Installing glue [1.6.1] ...
#>  OK [linked cache]
#> Installing jsonlite [1.7.3] ...
#>  OK [linked cache]
#> Installing magrittr [2.0.2] ...
#>  OK [linked cache]
#> Installing yaml [2.2.2] ...
#>  OK [linked cache]
#> Installing knitr [1.37] ...
#>  OK [linked cache]
#> Installing renv [0.15.2] ...
#>  OK [linked cache]
#> Installing rlang [1.0.0] ...
#>  OK [linked cache]
#> Installing tinytex [0.36] ...
#>  OK [linked cache]
#> Installing rmarkdown [2.11] ...
#>  OK [linked cache]
#> The following package(s) have been updated:
#> 
#>  glue [installed version 1.6.1  != loaded version 1.6.2 ]
#>  yaml [installed version 2.2.2  != loaded version 2.3.5 ]
#>  renv [installed version 0.15.2 != loaded version 0.15.4]
#> 
#> Consider restarting the R session and loading the newly-installed packages.
#> 
#> → Recording changes in lockfile
#> The following package(s) have unsatisfied dependencies:
#> 
#>  bookdown requires rmarkdown (>= 2.13), but version 2.11 is installed
#> 
#> Consider updating the required dependencies as appropriate.
#> 
#> Warning message:
#> This project is configured to use R version '4.1.2', but '4.2.0' is currently being used.
#> Error: callr subprocess failed: aborting snapshot due to pre-flight validation failure

^{Created on 2022-05-09 by the reprex package (v2.0.1)}

Winning Solution: pin_version() followed by manage_deps():

This worked because this process manually modifies the lockfile without having {renv} check it.

Winning solution that shows snapshot being written

sandpaper::pin_version("[email protected]", path = tmp)
#> * Updated 1 record in "/tmp/Rtmpa7Yb7x/file2831329580f12/renv/profiles/lesson-requirements/renv.lock".
sandpaper::manage_deps(tmp)
#> ℹ Consent to use package cache provided
#> → Searching for and installing available dependencies
#> Finding R package dependencies ... Done!
#> → Restoring any dependency versions
#> The following package(s) will be updated:
#> 
#> # Repository =========================
#> - rmarkdown   [repo: CRAN -> *; ver: 2.11 -> 2.14]
#> 
#> renv tried to install 'xfun 0.29', but the following constraints were not met:
#> 
#>  rmarkdown (requires xfun >= 0.30)
#> 
#> renv will try to install 'xfun 0.30' instead.
#> 
#> Installing xfun [0.30] ...
#>  OK [linked cache]
#> Installing rmarkdown [2.14] ...
#>  OK [linked cache]
#> The dependency tree was repaired during package installation:
#>        _
#>   xfun   [0.30]
#> 
#> Call `renv::snapshot()` to capture these dependencies in the lockfile.
#> 
#> → Recording changes in lockfile
#> The following package(s) will be updated in the lockfile:
#> 
#> # CRAN ===============================
#> - xfun       [0.29 -> 0.30]
#> - R6         [* -> 2.5.1]
#> - bookdown   [* -> 0.26]
#> - bslib      [* -> 0.3.1]
#> - fs         [* -> 1.5.2]
#> - rappdirs   [* -> 0.3.3]
#> - sass       [* -> 0.4.1]
#> 
#> * Lockfile written to '/tmp/Rtmpa7Yb7x/file2831329580f12/renv/profiles/lesson-requirements/renv.lock'.
#> Warning message:
#> This project is configured to use R version '4.1.2', but '4.2.0' is currently being used.

^{Created on 2022-05-09 by the reprex package (v2.0.1)}

Moving Forward

I think primarily, we need to tell renv to ignore the output yaml option or strip it from its view. We want people to bring in files from other contexts like bookdown, so it's reasonable to expect yaml headers that allow for that.

The problem with the pin_version() solution is that it's not guaranteed to work because you end up chasing down broken dependencies (an earlier version of this had a broken {xfun} and I solved it by deleting the local cache).