heroku
CI is broken for community contributions
the CI jobs seems to be relying on github secrets to be present. But those secrets are not exposed in pull_request jobs for untrusted PRs (i.e. PRs coming from forks) to avoid the risk of a malicious PR leaking the secret. See https://github.com/heroku/heroku-buildpack-php/actions/runs/9665966313/job/26664478221?pr=738 for the result of such CI job.
This creates a confusing experience for contributors.
Well, maybe those CI jobs requiring secrets should be skipped when secrets are not available instead of reporting weird failures.
I'm unable to see the output of that job I see "The logs for this run have expired and are no longer available." I'm curious to see what the experience is like if someone can share a screenshot, but I also trust that it's not a great experience.
what do you suggest we do about it? ;)
I suggest that we let people know ahead of time what to expect. One option could be adding a PR template that has explicit instructions/warnings around tests.
Well, maybe those CI jobs requiring secrets should be skipped when secrets are not available instead of reporting weird failures.
Unfortunately, most all tests require secrets due to the nature of what we're building and the testing strategy (the integration tests create real heroku apps and deploy them). We also have to be wary of compliance issues. We have need to make sure that our tests execute and are green before merge (or otherwise have a compliance exception). Skipping tests conditionally would bypass this.
I'm the maintainer of the Ruby buildpack and when I get a PR I have to push it back up as a branch within the repo so that tests will execute after inspecting it. It's quite a lot of work. Thankfully the CNB test suites test using containers and don't require secrets so it's much easier to accept a contribution there https://github.com/heroku/buildpacks-php.
I'm going to close this out for now. If you want to suggest a PR template with some words of warning about the process that would be appreciated. You can followup with me on https://ruby.social/@Schneems if no response (sometimes I miss emails even if you @-me on an issue).
