[feature request] a new `exclude` argument into `run_examples()`

[pawelru]

Is your feature request related to a problem? Please describe. I am thinking about testing examples outside of R CMD CHECK. This would allow me specify more strict options like warning on partial argument match, lifecycle warnings to errors, even all warnings to errors and things like that. I could also try to use mocking in donttest and dontrun ones. These are just ideas right now that needs further refinement.

Describe the solution you'd like Very likely I would encounter exceptions down the way so I would like to be able to exclude individual Rd files (or even individual examples if possible). Therefore I would need an exclude argument to the run_examples() function. I feel it would give much better control than existing start argument.

Describe alternatives you've considered An alternative is to get .Rd files using non-exported devtools function, apply exclusions and then test individual examples using pkgload::run_example().

withr::with_options(
  <my strict options>,
  {
    files <- devtools:::rd_files()
    exclude <- c(
      "foo.Rd",
      "bar.Rd"
    )
    files <- files[!basename(files) %in% exclude]
    for (i in files) pkgload::run_example(i, run_donttest = TRUE, run_dontrun = FALSE)
  }
)

Additional context Add any other context or about the feature request here.

[jennybc]

I'm not working on devtools at the moment, but made a quick visit while clearing notifications.

One high-level observation I have about this issue and #2545 is that it feels like the participants are sort of approaching examples as if they are tests. The tidyverse team maintains a lot (>200) packages and I'm pretty sure none of us make regular usage of run_examples(). Now that doesn't mean it's a bad function! Rather I think we approach examples differently. When tinkering with them, we execute them interactively. Then, if unexpected problems crop up, we're going to hear about it in CI, i.e. one of our R CMD check or pkdgown jobs is going to fail.

A lot of the features being requested are available already in the testing side of things and it just feels like the code where you need all of that control sounds more like test code than example code.

[orgadish]

@jennybc I have found that I use run_examples() / run_example() a lot when first developing my packages, i.e. when I'm still working out the kinks in new functionality. You're right that it's sort of like testing, but since the examples also need to be debugged at the beginning, having more control over what you run can be very helpful.

Having said that, I do think pkgload::run_example(path, ...) and pkgload::dev_example(topic, ...) cover what I need (and allow for the kind of functionality that @pawelru is referring). In #2545, I just hadn't realized those existed (I've since updated that request for clarity).