Tooling to support CRAN submission with known breakages

[hadley]

CRAN would appreciate it if we reported known failures in a standard format. We need both a human readable text format (for inclusion in the email that CRAN team can easily scan) and something machine readable that they can easily compute with (e.g. attached .rds)

Package: chunked
Check: R code for possible problems
New result: ERROR
Explanation: patch at https://github.com/edwindj/chunked/pull/21

Package: functiondepends
Check: re-building of vignette outputs
New result: WARNING
Explanation: patch at https://github.com/jakubsob/functiondepends/pull/2

We'll need to think about how we should track the needed data from running the revdeps, to tracking progress on issues/PRs, to final submission.

[jennybc]

Seems like it will require coordination with revdepcheck.

[hadley]

There's a little bit of coordination required, but the most important bit here is adding some standardised way for us to capture our responses to a revdepcheck failure.

[jennybc]

Ah, so also seems related to potentially standardising the human bit, where we track a set of revdepcheck failures in a spreadsheet or similar.

[DavisVaughan]

Anecdotal conversation with CRAN that can guide some of this:

We sent in a workflows submission with 3 known breakages. This was cran-comments.md:

## revdepcheck results

We checked 28 reverse dependencies (26 from CRAN + 2 from Bioconductor), comparing R CMD check results across CRAN and dev versions of this package.

 * We saw 3 new problems
 * We failed to check 0 packages

Issues with CRAN packages are summarised below.

### New problems
(This reports the first line of each new failure)

* modeltime
  checking tests ... ERROR

* modeltime.ensemble
  checking tests ... ERROR

* modeltime.resample
  checking re-building of vignette outputs ... WARNING

CRAN responded:

Nice you found these, too, but this is un helpful: Is this expected?
Have the maintainers been informed well in advance and when?

We actually had done a PR about this, but our standard tooling didn't give us a way to let CRAN know about it https://github.com/business-science/modeltime/pull/198

I responded further, asking if we could improve our process:

Is there a form of `cran-comments.md` that we (tidyverse, r-lib, tidymodels) could send you that would be most helpful to you and CRAN? Possibly something that is machine readable in some way for easy processing?

To which CRAN replied:

No. Ideally only information we need. In your case: Which rev deps
failed and what have you done to inform the maintainers.

So which nonstandard things have to be coinsidered that may prevent auto
publication and are supposed to be fine. And which additional issues
have  been fixed.


Machine readable does not help here, as we have to inspect the package
manually if auto publication was not possible. Ideally I do not want to
read what our checks will find anyway.
Most people include cran comment files with information from former
submissions, which is also irrelevant for us.

Honestly, I hate these cran-comments files, and would like to see the
comments that people want to tell us for the new submission. So what
people who do not use devtools would simply enter in the webform.

---

So it seems like:

• We should not include historical information in cran-comments.md
• They don't want a machine readable form
• We should list:
  • Known failures
  • The date we informed the maintainers that we were going to break their package
  • Any links to PRs / issues we opened for each broken package
• Possibly include a summary of what is being broken

[hadley]

I think we should update this for our latest workflow: managing a separate issue of all breakages and links to PRs. That seems to be working well for us.

[DavisVaughan]

My plan for cran-comments.md going forward was going to be:

• Mention how many broke. Tell CRAN we have sent PRs with at least 2 weeks notice.
• Link them to the issue where we manage breakage and PR links

This seems to hit the highlights of what CRAN told me they actually care about, while being minimal extra work for us

[DavisVaughan]

Here is what we are doing in cran-comments.md for tidyr 1.3.0 https://github.com/tidyverse/tidyr/blob/69b74ef967b6170fe2346c71236996f0a8e6347f/cran-comments.md