Support full `.gitignore` syntax in `.rcsignore`

[ColinFay]

Issue

The . rcsignore file doesn't work with :

1. folders that are listed as folder/ instead of folder
2. files that are in subdirectories

Diagnosis

1. For dev/ : the setdiff here is not smart enough : https://github.com/rstudio/rsconnect/blob/cf0972d0d69357cf2bc81ab140f940085aac92cb/R/bundleFiles.R#L200

list.files() (that created the contents object) will do

[1] "DESCRIPTION" "dev"         "inst"        "man"         "NAMESPACE"  
[6] "R" 

When a directory can usually be written folder/ in an ignore file.

2. The list file that creates the contents object doesn't list subfolders: recursiveBundleFiles doesn't do a recursive in the list file https://github.com/rstudio/rsconnect/blob/cf0972d0d69357cf2bc81ab140f940085aac92cb/R/bundleFiles.R#L137

How to reproduce

1. Go to /tmp & create a golem app:

; cd /tmp && Rscript -e 'golem::create_golem("reprexrsconnect")' && cd reprexrsconnect

4. Add 'DESCRIPTION' to .rscignore => works as expected

; echo 'DESCRIPTION' >> .rscignore  
; ls
DESCRIPTION     NAMESPACE       R               dev             inst            man
; Rscript -e 'rsconnect::listDeploymentFiles(".")'
 [1] ".here"                    ".Rbuildignore"           
 [3] "dev/01_start.R"           "dev/02_dev.R"            
 [5] "dev/03_deploy.R"          "dev/run_dev.R"           
 [7] "inst/app/www/favicon.ico" "inst/golem-config.yml"   
 [9] "man/run_app.Rd"           "NAMESPACE"               
[11] "R/app_config.R"           "R/app_server.R"          
[13] "R/app_ui.R"               "R/run_app.R"

5. Trying to ignore dev/ => doesn't work

; echo 'dev/' >> .rscignore
; Rscript -e 'rsconnect::listDeploymentFiles(".")'
 [1] ".here"                    ".Rbuildignore"           
 [3] "dev/01_start.R"           "dev/02_dev.R"            
 [5] "dev/03_deploy.R"          "dev/run_dev.R"           
 [7] "inst/app/www/favicon.ico" "inst/golem-config.yml"   
 [9] "man/run_app.Rd"           "NAMESPACE"               
[11] "R/app_config.R"           "R/app_server.R"          
[13] "R/app_ui.R"               "R/run_app.R"      

6. Trying to ignore dev/01_start.R => doesn't work

 ;echo 'dev/01_start.R' >> .rscignore 
 [1] ".here"                    ".Rbuildignore"           
 [3] ".rscignor"                "dev/01_start.R"          
 [5] "dev/02_dev.R"             "dev/03_deploy.R"         
 [7] "dev/run_dev.R"            "inst/app/www/favicon.ico"
 [9] "inst/golem-config.yml"    "man/run_app.Rd"          
[11] "NAMESPACE"                "R/app_config.R"          
[13] "R/app_server.R"           "R/app_ui.R"              
[15] "R/run_app.R"             

7. Printing the rscignore just to be sure

; cat .rscignore 
DESCRIPTION
dev/
dev/01_start.R

Happy to help with submitting a patch if you want :)

[hadley]

Do you mind creating a reprex for me? The following code looks correct to me.

library(rsconnect)

dir <- withr::local_tempdir()
dir.create(file.path(dir, "a", "b"), recursive = TRUE)
file.create(file.path(dir, c("x", "a/y", "a/b/z")))
#> [1] TRUE TRUE TRUE
listDeploymentFiles(dir)
#> [1] "a/b/z" "a/y"   "x"

writeLines("y", file.path(dir, "a", ".rscignore"))
listDeploymentFiles(dir)
#> [1] "a/b/z" "x"

writeLines(c("y", "b"), file.path(dir, "a", ".rscignore"))
listDeploymentFiles(dir)
#> [1] "x"

^{Created on 2023-03-16 with reprex v2.0.2}

[hadley]

Or are you asking about this?

#' *  You can exclude additional files by listing them in in a `.rscignore`
#'    file. This file must have one file or directory per line (with path
#'    relative to the current directory). It doesn't support wildcards, or
#'    ignoring files in subdirectories.

[hadley]

This is not technically supported, but it's a popular request, so I'll bite the bullet and implement it. See implementation in renv for a starting point: https://github.com/rstudio/renv/blob/main/R/renvignore.R.

Looking forward, should read both .rscignore and .connectignore.

[karawoo]

We're looking into broadening the syntax of .rscignore to match that of .gitignore (and .renvignore). This would include support for trailing slashes in directories, files in subdirectories, and wildcards. I wanted to examine how the behavior of existing .rscignore files in the wild would change. Attaching my script below for posterity. The tl;dr:

• If .rscignore follows the same syntax as .renvignore/.gitignore it may result in ignoring more files than are currently ignored. For example, putting app.R in .rscignore will ignore a file called app.R in that directory only. Under the new ignore rules, any file called app.R in that directory or any child directory will be ignored. People would need to update their .rscignore to add a leading / if they want to maintain the existing .rscignore behavior.
• I haven't found any cases where files that are currently ignored by .rscignore would become un-ignored.

This has the potential to be a breaking change as new files might get ignored and cause deployed content to break because they're now missing from the bundle, however I think it's a change worth making.

script.R

## This script compares the behavior of our existing .rscignore files with
## .renvignore/.gitignore files which support wildcards and subdirectories

## makes use of the local_temp_app() helper in rsconnect. update with the
## appropriate path:
devtools::load_all("rsconnect")

## Create a project with various subdirectories containing files (which are
## empty for our purposes)

## project
## ├── app.R
## ├── apple
## │   ├── app.R
## │   └── banana
## │       └── cherry
## │           └── app.R
## ├── apple.R
## └── cherry
## └── app.R

base_project <- list(
  "app.R" = "#",
  "apple.R" = "#",
  "apple/app.R" = "#",
  "apple/banana/cherry/app.R" = "#",
  "cherry/app.R" = "#"
)

test_cases <- list(
  # .rscignore in root dir --------------------------------------------------

  ## Ignoring a file
  c(
    base_project,
    ".rscignore" = "app.R"
  ),
  ## Ignoring a file from the root of the project specifically
  c(
    base_project,
    ".rscignore" = "/app.R"
  ),
  ## Ignoring a directory
  c(
    base_project,
    ".rscignore" = "cherry"
  ),
  ## Ignoring a directory with trailing slash
  c(
    base_project,
    ".rscignore" = "cherry/"
  ),
  ## Ignoring a directory with leading and trailing slash
  c(
    base_project,
    ".rscignore" = "/cherry/"
  ),
  ## Wildcard
  c(
    base_project,
    ".rscignore" = "app*.R"
  ),

  # .rscignore in subdir ----------------------------------------------------

  c(
    base_project,
    "apple/.rscignore" = "cherry"
  ),
  c(
    base_project,
    "apple/.rscignore" = "cherry/"
  ),
  c(
    base_project,
    "apple/.rscignore" = "/cherry/"
  ),

  # multiple .rscignore files -----------------------------------------------

  c(
    base_project,
    ".rscignore" = "app.R",
    "cherry/.rscignore" = "app.R"
  )
)


## Returns files ignored by rsconnect when creating bundle
rsc_ignored <- function(project) {
  app_dir <- suppressWarnings(local_temp_app(project))
  setdiff(list.files(app_dir, recursive = TRUE), rsconnect:::bundleFiles(app_dir))
}

## Returns files ignored by renv
renv_ignored <- function(project) {
  app_dir <- suppressWarnings(local_temp_app(project))
  included <- renv:::renv_dependencies_find_dir(app_dir, app_dir, 0)
  included <- unlist(included)
  included <- gsub(paste0(app_dir, "/"), "", included)
  setdiff(list.files(app_dir, recursive = TRUE), included)
}

## Returns files ignored by git. These should be the same as those ignored by
## renv, but just double checking
git_ignored <- function(project) {
  app_dir <- suppressWarnings(local_temp_app(project))
  system(glue::glue("git -C {app_dir} init"), intern = TRUE)
  system(glue::glue("git -C {app_dir} add ."), intern = TRUE)
  tracked <- system(glue::glue("git -C {app_dir} ls-files"), intern = TRUE)
  setdiff(list.files(app_dir, recursive = TRUE), tracked)
}

## Compares the behavior of rsconnect, renv, and git
compare_behavior <- function(project) {
  renv_proj <- project
  names(renv_proj) <- gsub("\\.rscignore$", ".renvignore", names(renv_proj))

  git_proj <- project
  names(git_proj) <- gsub("\\.rscignore$", ".gitignore", names(git_proj))

  ignore_files <- project[grepl("\\.rscignore$", names(project))]

  results <- list(
    `ignore files` = ignore_files,
    rsconnect = rsc_ignored(project),
    renv = renv_ignored(renv_proj),
    git = git_ignored(git_proj)
  )
  results
}

(comparisons <- lapply(test_cases, compare_behavior))

## Are there any cases where changing syntax would result in the inclusion of
## files that are currently being ignored by rsconnect?
new_inclusions <- function(comp) {
  length(setdiff(comp$rsconnect, comp$renv)) > 0 || length(setdiff(comp$rsconnect, comp$git)) > 0
}

any(sapply(comparisons, new_inclusions))
#> [1] FALSE