remotes
remotes copied to clipboard
r-lib
Remotes fails with private package dependency
I think this is the overall issue similar to #701 and the same issue as #697.
Description: If you pass in auth_token to install_github and don't have GITHUB_PAT set (so that remotes:::github_pat() returns NULL), you will get a failure. Let's say we have pkgA (private repo, muschellij2/pkgA) and pkgB (private repo, muschellij2/pkgB), that depends on pkgA and has pkgA set in the Remotes field.
Below, I have a reprex that demonstrates the issue, but the issue is that the packages are private (which is the overall issue). I'm happy to make an r-lib maintainer to make things easier. I also created a video demonstrating the issue: https://youtu.be/qZi_50QojbM.
Issue: Expected behavior is that if auth_token is passed to install_github, then GITHUB_PAT does not need to be set. The issue is that auth_token doesn't get passed to the dependency remotes. The failure happens at remote_sha, and package2remote (as package2remote uses remotes:::github_pat() for the auth_token parameter).
Solution: If you have private dependencies, you must set GITHUB_PAT for this to work. The auth_token will not pass through.
Reprex
GITHUB_PAT is in object auth_token - done outside of reprex (deleted line)
nchar(auth_token)
#> [1] 40
Demonstration of the issue
Overall, things are fine if GITHUB_PAT or GITHUB_TOKEN are set as environment variables, because remotes::github_pat will pick them up. In our case, we see they are not set:
Sys.getenv("GITHUB_PAT")
#> [1] ""
Sys.getenv("GITHUB_TOKEN")
#> [1] ""
Try to install package - fails because pkgA is private
remotes::install_github("muschellij2/pkgA")
#> Error: Failed to install 'unknown package' from GitHub:
#> HTTP error 404.
#> Not Found
#>
#> Did you spell the repo owner (`muschellij2`) and repo name (`pkgA`) correctly?
#> - If spelling is correct, check that you have the required permissions to access the repo.
Pass in auth_token directly - succeeds
remotes::install_github("muschellij2/pkgA", auth_token = auth_token)
#> Downloading GitHub repo muschellij2/pkgA@HEAD
#> * checking for file ‘/tmp/RtmpzJUpNR/remotes1acb477ff0ad/muschellij2-pkgA-9d94302887cda58639c76f1e06ffc3664e29138d/DESCRIPTION’ ... OK
#> * preparing ‘pkgA’:
#> * checking DESCRIPTION meta-information ... OK
#> * checking for LF line-endings in source and make files and shell scripts
#> * checking for empty or unneeded directories
#> * building ‘pkgA_0.0.0.9000.tar.gz’
#> Installing package into '/home/gcer/R'
#> (as 'lib' is unspecified)
Following fails because pkgA is private and auth_token passed. But note that it says Failed to install 'pkgB' from GitHub:
remotes::install_github("muschellij2/pkgB", auth_token = auth_token)
#> Downloading GitHub repo muschellij2/pkgB@HEAD
#> Error: Failed to install 'pkgB' from GitHub:
#> HTTP error 404.
#> Not Found
#>
#> Did you spell the repo owner (`muschellij2`) and repo name (`pkgA`) correctly?
#> - If spelling is correct, check that you have the required permissions to access the repo.
Issue resolved if GITHUB_PAT is set
Here we will set GITHUB_PAT, show remotes:::github_pat picks it up, and install_github works fine.
Sys.getenv("GITHUB_PAT")
#> [1] ""
Sys.setenv(GITHUB_PAT = auth_token)
nchar(remotes:::github_pat())
#> [1] 40
The installation succeeds below because it is using GITHUB_PAT for both auth_token (initial package call) and when remotes:::github_pat() called in dependency.
remotes::install_github("muschellij2/pkgB")
#> Using github PAT from envvar GITHUB_PAT. Use `gitcreds::gitcreds_set()` and unset GITHUB_PAT in .Renviron (or elsewhere) if you want to use the more secure git credential store instead.
#> Downloading GitHub repo muschellij2/pkgB@HEAD
#>
#> * checking for file ‘/tmp/RtmpzJUpNR/remotes1acb60a90f49/muschellij2-pkgB-c3b57cff32842cd365e1d5582f981648a28e7705/DESCRIPTION’ ... OK
#> * preparing ‘pkgB’:
#> * checking DESCRIPTION meta-information ... OK
#> * checking for LF line-endings in source and make files and shell scripts
#> * checking for empty or unneeded directories
#> * building ‘pkgB_0.0.0.9000.tar.gz’
#> Installing package into '/home/gcer/R'
#> (as 'lib' is unspecified)
remove.packages("pkgB")
#> Removing package from '/home/gcer/R'
#> (as 'lib' is unspecified)
This includes succeeding when auth_token is passed (auth_token used for pkgB, remotes:::github_pat used for pkgA and all dependencies:
remotes::install_github("muschellij2/pkgB", auth_token = auth_token)
#> Downloading GitHub repo muschellij2/pkgB@HEAD
#>
#> * checking for file ‘/tmp/RtmpzJUpNR/remotes1acb2efeb5e2/muschellij2-pkgB-c3b57cff32842cd365e1d5582f981648a28e7705/DESCRIPTION’ ... OK
#> * preparing ‘pkgB’:
#> * checking DESCRIPTION meta-information ... OK
#> * checking for LF line-endings in source and make files and shell scripts
#> * checking for empty or unneeded directories
#> * building ‘pkgB_0.0.0.9000.tar.gz’
#> Installing package into '/home/gcer/R'
#> (as 'lib' is unspecified)
remove.packages("pkgB")
#> Removing package from '/home/gcer/R'
#> (as 'lib' is unspecified)
Sys.unsetenv("GITHUB_PAT")
Demonstrating it is GITHUB_PAT
Here can can pass in garbage to GITHUB_PAT to demonstrate the main repo (pkgB) is using auth_token by getting information. Note below says Failed to install 'pkgB' from GitHub:
Sys.setenv(GITHUB_PAT = "asdlfkjas;dfja")
remotes::install_github("muschellij2/pkgB", auth_token = auth_token)
#> Downloading GitHub repo muschellij2/pkgB@HEAD
#> Error: Failed to install 'pkgB' from GitHub:
#> HTTP error 401.
#> Bad credentials
#>
#> Rate limit remaining: 53/60
#> Rate limit reset at: 2023-06-02 19:06:18 UTC
#>
#>
remove.packages("pkgB")
#> Removing package from '/home/gcer/R'
#> (as 'lib' is unspecified)
#> Error in find.package(pkgs, lib): there is no package called 'pkgB'
Sys.unsetenv("GITHUB_PAT")
Here can can pass in garbage to auth_token to show that it fails at the point of reading the main package, such that it is not using GITHUB_PAT for the main repo read. Note unknown package vs. pkgB above:
Sys.getenv("GITHUB_PAT")
#> [1] ""
Sys.setenv(GITHUB_PAT = auth_token)
remotes::install_github("muschellij2/pkgB", auth_token = "asdfdsf")
#> Error: Failed to install 'unknown package' from GitHub:
#> HTTP error 401.
#> Bad credentials
#>
#> Rate limit remaining: 52/60
#> Rate limit reset at: 2023-06-02 19:06:18 UTC
#>
#>
remove.packages("pkgB")
#> Removing package from '/home/gcer/R'
#> (as 'lib' is unspecified)
#> Error in find.package(pkgs, lib): there is no package called 'pkgB'
Sys.unsetenv("GITHUB_PAT")
Where the problem occurs
The issue occurs in dev_package_deps. In order to use dev_package_deps locally, we need to clone the package, and need to set GITHUB_PAT for git2r::cred_token:
Sys.getenv("GITHUB_PAT")
#> [1] ""
Sys.setenv(GITHUB_PAT = auth_token)
pkgdir = tempfile()
git2r::clone(url = "https://github.com/muschellij2/pkgB",
local_path = pkgdir,
credentials = git2r::cred_token())
#> cloning into '/tmp/RtmpzJUpNR/file1acb67656799'...
#> Receiving objects: 8% (1/12), 9 kb
#> Receiving objects: 16% (2/12), 9 kb
#> Receiving objects: 25% (3/12), 9 kb
#> Receiving objects: 33% (4/12), 9 kb
#> Receiving objects: 41% (5/12), 9 kb
#> Receiving objects: 58% (7/12), 9 kb
#> Receiving objects: 66% (8/12), 13 kb
#> Receiving objects: 75% (9/12), 13 kb
#> Receiving objects: 83% (10/12), 13 kb
#> Receiving objects: 91% (11/12), 13 kb
#> Receiving objects: 100% (12/12), 13 kb, done.
#> Local: main /tmp/RtmpzJUpNR/file1acb67656799
#> Remote: main @ origin (https://github.com/muschellij2/pkgB)
#> Head: [c3b57cf] 2023-06-02: added things
We can see that it does work:
readLines(file.path(pkgdir, "DESCRIPTION"))
#> [1] "Package: pkgB"
#> [2] "Title: What the Package Does (One Line, Title Case)"
#> [3] "Version: 0.0.0.9000"
#> [4] "Authors@R: "
#> [5] " person(\"First\", \"Last\", , \"[email protected]\", role = c(\"aut\", \"cre\"),"
#> [6] " comment = c(ORCID = \"YOUR-ORCID-ID\"))"
#> [7] "Description: What the package does (one paragraph)."
#> [8] "License: GPL (>= 3)"
#> [9] "Encoding: UTF-8"
#> [10] "Roxygen: list(markdown = TRUE)"
#> [11] "RoxygenNote: 7.2.3"
#> [12] "Imports: "
#> [13] " pkgA"
#> [14] "Remotes:"
#> [15] " muschellij2/pkgA"
Pulling package deps succeeds because GITHUB_PAT set
r = remotes::dev_package_deps(pkgdir = pkgdir)
But dev_package_deps fails after we unset GITHUB_PAT
Sys.unsetenv("GITHUB_PAT")
remotes::dev_package_deps(pkgdir = pkgdir)
#> Error: HTTP error 404.
#> Not Found
#>
#> Did you spell the repo owner (`muschellij2`) and repo name (`pkgA`) correctly?
#> - If spelling is correct, check that you have the required permissions to access the repo.
Digging into dev_package_deps
Code extracted from dev_package_deps:
# setting up vars like defaults in `dev_package_deps`
repos = getOption("repos")
dependencies = NA
type = getOption("pkgType")
pkg <- remotes:::load_pkg_description(pkgdir)
repos <- c(repos, remotes:::parse_additional_repositories(pkg))
deps <- remotes:::local_package_deps(pkgdir = pkgdir, dependencies = dependencies)
deps
#> [1] "pkgA"
In dev_package_deps, the code fails at package_deps (https://github.com/r-lib/remotes/blob/e199c1bdda3858600bbef8e4bbc3f5868b899587/R/deps.R#L143)
cran_deps <- remotes:::package_deps(deps, repos = repos, type = type)
#> Error: HTTP error 404.
#> Not Found
#>
#> Did you spell the repo owner (`muschellij2`) and repo name (`pkgA`) correctly?
#> - If spelling is correct, check that you have the required permissions to access the repo.
Digging into remotes:::package_deps
The variables change names a bit, but we pass deps into the packages argument for remotes:::package_deps
packages = deps
repos <- remotes:::fix_repositories(repos)
cran <- remotes:::available_packages(repos, type)
deps <- remotes:::find_deps(packages, available = cran, top_dep = dependencies)
deps
#> [1] "pkgA"
Here, we see no auth_token is passed as package2remote uses github_pat:
https://github.com/r-lib/remotes/blob/e199c1bdda3858600bbef8e4bbc3f5868b899587/R/install-remote.R#L241
remote <- structure(lapply(deps, remotes:::package2remote, repos = repos, type = type), class = "remotes")
Without GITHUB_PAT set, we do not get the auth needed:
remote[[1]]$auth_token
#> NULL
And setting GITHUB_PAT, it gets correctly passed to remote:
Sys.setenv(GITHUB_PAT = auth_token)
remote <- structure(lapply(deps, remotes:::package2remote, repos = repos, type = type), class = "remotes")
nchar(remote[[1]]$auth_token)
#> [1] 40
Created on 2023-06-02 with reprex v2.0.2
Session info
sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#> setting value
#> version R version 4.2.2 (2022-10-31)
#> os Ubuntu 22.04.1 LTS
#> system x86_64, linux-gnu
#> ui X11
#> language (EN)
#> collate en_US.UTF-8
#> ctype en_US.UTF-8
#> tz Etc/UTC
#> date 2023-06-02
#> pandoc 2.19.2 @ /usr/lib/rstudio-server/bin/quarto/bin/tools/ (via rmarkdown)
#>
#> ─ Packages ───────────────────────────────────────────────────────────────────
#> package * version date (UTC) lib source
#> callr 3.7.3 2022-11-02 [2] RSPM (R 4.2.0)
#> cli 3.6.1 2023-03-23 [1] RSPM (R 4.2.0)
#> crayon 1.5.2 2022-09-29 [1] RSPM
#> curl 5.0.0 2023-01-12 [1] RSPM (R 4.2.0)
#> digest 0.6.31 2022-12-11 [1] RSPM (R 4.2.0)
#> evaluate 0.19 2022-12-13 [1] RSPM (R 4.2.0)
#> fastmap 1.1.1 2023-02-24 [1] RSPM
#> fs 1.6.1 2023-02-06 [1] RSPM (R 4.2.0)
#> gcloud 0.6.0 2023-04-27 [1] local
#> git2r 0.30.1 2022-03-16 [1] RSPM (R 4.2.0)
#> glue 1.6.2 2022-02-24 [1] RSPM
#> highr 0.10 2022-12-22 [1] RSPM (R 4.2.0)
#> htmltools 0.5.4 2022-12-07 [1] RSPM (R 4.2.0)
#> jsonlite 1.8.4 2022-12-06 [1] RSPM (R 4.2.0)
#> knitr 1.41 2022-11-18 [1] RSPM (R 4.2.0)
#> lifecycle 1.0.3 2022-10-07 [1] RSPM
#> magrittr 2.0.3 2022-03-30 [1] RSPM
#> pkgbuild 1.3.1 2021-12-20 [2] RSPM (R 4.2.0)
#> prettyunits 1.1.1 2020-01-24 [2] RSPM (R 4.2.0)
#> processx 3.8.0 2022-10-26 [2] RSPM (R 4.2.0)
#> ps 1.7.2 2022-10-26 [2] RSPM (R 4.2.0)
#> R6 2.5.1 2021-08-19 [1] RSPM
#> remotes 2.4.2.9000 2023-06-02 [1] local
#> reprex 2.0.2 2022-08-17 [2] RSPM (R 4.2.0)
#> rlang 1.1.0 2023-03-14 [1] RSPM
#> rmarkdown 2.18 2022-11-09 [1] RSPM (R 4.2.0)
#> rprojroot 2.0.3 2022-04-02 [2] RSPM (R 4.2.0)
#> rstudioapi 0.14 2022-08-22 [2] RSPM (R 4.2.0)
#> sessioninfo 1.2.2.9000 2022-11-19 [1] Github (r-lib/sessioninfo@0d74fe9)
#> stringi 1.7.12 2023-01-11 [1] RSPM (R 4.2.0)
#> stringr 1.5.0 2022-12-02 [1] RSPM (R 4.2.0)
#> vctrs 0.6.2 2023-04-19 [1] RSPM (R 4.2.0)
#> withr 2.5.0 2022-03-03 [1] RSPM
#> xfun 0.36 2022-12-21 [1] RSPM (R 4.2.0)
#> yaml 2.3.6 2022-10-18 [2] RSPM (R 4.2.0)
#>
#> [1] /home/gcer/R
#> [2] /usr/local/lib/R/site-library
#> [3] /usr/local/lib/R/library
#>
#> ──────────────────────────────────────────────────────────────────────────────
Thanks for the detailed report! I suggest you try the pak package for this, it might work out of the box.
