Tokenless support in the new CLI (for public repos and forks)

[toddbaert]

Hello! I've recently updated a number of projects in our org to use the latest versions of the codecov action.

I see now that the codecov CLI is used to power this action, and that it requires a token. No problem, I went and got the token for the repositories in question, and all seemed to work... until we got a PR from a fork (something that happens quite frequently in our case). This results in a problem: secrets aren't available in PRs from forks (unless we use the pull_request_target trigger, which would allow malicious actors to potentially exfiltrate our secret(s)).

So my question is, as an open-source project, how am I supposed to use the new action? I want to run the coverage analysis on the new code (which is untrusted) but in order to do that, I need to run in a trusted environment to get access to the secret. Am I missing something? :thinking:

[thomasrockhu-codecov]

@toddbaert this is a known and yet unsolved problem. We are investigating GitHub's OIDC to see if this will be a suitable replacement for token-less uploading (including forks)

[toddbaert]

@thomasrockhu-codecov Thanks!

For now we'll stick to v3! Happy to help you folks test v4 if you have a proposed solution for forks at some point.

Should I close this issue?

[thomasrockhu-codecov]

@toddbaert let's leave it open for now so that anyone watching will get an update if OIDC is the way to go for forks.

For everyone else, you can add the CODECOV_TOKEN from the Codecov repo settings page to your repo secrets. Then, update your GitHub Actions workflows:

...
uses: codecov/codecov-action@v4
env:
  CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
...

[toddbaert]

@toddbaert let's leave it open for now so that anyone watching will get an update if OIDC is the way to go for forks.

For everyone else, you can add the CODECOV_TOKEN from the Codecov repo settings page to your repo secrets. Then, update your GitHub Actions workflows:

...
uses: codecov/codecov-action@v4
env:
  CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
...

Yep, looks like there's some linked PRs/issues, so probably good to keep it open until there's a solution for forks. Thanks again!

[dolfinus]

you can add the CODECOV_TOKEN from the Codecov repo settings page to your repo secrets

But this still requires changing on:pull_request to on:pull_request_target to access secrets of original repo from a fork.

[rohan-at-sentry]

Hey all, thanks for this discussion. We're actively looking to solve this issue - https://github.com/codecov/engineering-team/issues/665

I'll post back here when we're ready! Thanks for your patience.

[jonhoo]

@rohan-at-sentry Now that https://github.com/codecov/engineering-team/issues/665 has been resolved, what's the state of affairs for this (and https://github.com/codecov/codecov-action/issues/1110)?

[rohan-at-sentry]

@jonhoo we're still waiting on a few related stories to wrap up, specifically around our API, following which we intend to bump up the version of the action.

I'll check back in here when this is done, but I expect things to wrap up fairly quick here

[jonhoo]

Nice, I see https://github.com/codecov/engineering-team/issues/736 was just completed! Steady steps forward :tada:

[rohan-at-sentry]

That's right. We actually released this under a flag that allowed us to test things out. Next steps are

1. Remove the flag
2. Update our GitHub Action, Bitrise Step and CircleCI orb

[rohan-at-sentry]

Hi everyone

We've released a new version of the codecov action and CLI that should support tokenless uploads from forks. Here's a link to our blog outlining the latest update including how to upgrade to the CLI. You can also visit our docs for more detailed information on how to get set up with the CLI, especially if you're not using our Github Action, Bitrise Step or CircleCI orb.

Currently, tokenless uploading is only available to forks attempting to create PRs on public repositories. This means that for most use cases, you will still need a token, even for public repositories. However, contributors creating PRs from their forks to the upstream repo will be able to get coverage information from Codecov.

[jrfnl]

@rohan-at-sentry If forks can get the codecov report without a token, why can't PRs from within the repo itself ? This sounds like the world upside-down.

[LecrisUT]

But PRs from forks to main repo have access to the main repo's secret so it will be using that token. The difference would be if they are making a PR on their own fork, in which case they should be uploading CODECOV_TOKEN to the forked repository as well

[thomasrockhu-codecov]

Hi all, to clear some things up about why tokenless is only available for forks to public repos.

The reason why tokenless has been so problematic over the past few years has been due to GitHub's rate limiting. Tokenless generally works by making an API call to GitHub to confirm that the repo and commit are the correct values. Making this call for thousands of repositories causes our GitHub token to hit the limit causing the issues that many of you have seen (e.g. Unable to locate build via Github Actions API).

As a result, we needed to make decisions on how best to serve our users. This means that tokenless uploading is not currently sustainable until GitHub allows for higher rate limited APIs.

The open source (OS) community is a major user and proponent of Codecov. A very common flow for them is to have outside contributors fork the repo, make changes, and open a PR to the upstream repository with that change. Unfortunately, GitHub does not have a way to share secrets with contributors. In order to provide support for these OS projects, we felt that it was necessary to allow for tokenless to forks.

We felt that this balance between low GitHub API usage and OS usability would allow for the least amount of pain for our users.

---

For forks using Codecov, you will need to use the fork's CODECOV_TOKEN when uploading to your own repository. However, when creating a PR upstream, that token will be ignored.

[jrfnl]

But PRs from forks to main repo have access to the main repo's secret so it will be using that token.

@LecrisUT No, they don't. They have access to the default GITHUB_TOKEN, but not to any repo secrets, which is what the CODECOV_TOKEN would be. (That's the whole point of this ticket)

[jrfnl]

Making this call for thousands of repositories causes our GitHub token to hit the limit causing the issues that many of you have seen (e.g. Unable to locate build via Github Actions API).

This means that tokenless uploading is not currently sustainable until GitHub allows for higher rate limited APIs.

That sounds like a "you" problem which you are dumping onto your end-users instead.

[thomasrockhu-codecov]

@jrfnl, I'm not sure what problem you're running into is. Public repos, yes, need to use a token, as do private repos to use Codecov. Public forks are now able to get coverage results back from Codecov on their PRs without the token. Are you having trouble adding the token?

[LecrisUT]

But PRs from forks to main repo have access to the main repo's secret so it will be using that token.

@LecrisUT No, they don't. They have access to the default GITHUB_TOKEN, but not to any repo secrets, which is what the CODECOV_TOKEN would be. (That's the whole point of this ticket)

I am confused then. How do tools like sonarcloud work on forks? random example. Basically isn't codecov supposed to mimic the same interface like those tools?

[Andrew-S-Rosen]

Thanks for the update, although it is an unfortunate one.

To confirm/clarify, in updating the codecov-action to 4.0, I am getting upload errors that no codecov token can be found. I was not getting this error with 3.0. Presumably the suggestion from the codecov team is to define my CODECOV_TOKEN in the GitHub Action, but this is obviously not practical when I have external contributors submitting PRs (which is kind of the whole point of open-source software in the first place).

[thomasrockhu-codecov]

@Andrew-S-Rosen, external contributors submitting PRs from forks without the token is allowed. Is this the use case you're talking about?

[thomasrockhu-codecov]

@LecrisUT I am unfamiliar with how sonarcloud works, so I can't speak on what API calls they need to do internally.

[Andrew-S-Rosen]

@Andrew-S-Rosen, external contributors submitting PRs from forks without the token is allowed. Is this the use case you're talking about?

Indeed it is. That would be great if so. However, I remain confused by some of the information here.

This means that for most use cases, you will still need a token, even for public repositories.

What are "most use cases" here then? I also was under the impression that repo secrets are not accessible from externally submitted PRs.

Is the suggestion to add

  env:
    token: ${{ secrets.CODECOV_TOKEN }}

where CODECOV_TOKEN is a defined repo secret. And then nothing else will need to be done? PRs submitted from forks will have their code coverage reported in the open PR as-is?

[mattlord]

It seems to be working for me here: https://github.com/vitessio/vitess/pull/14967

Thank you!

Now I have to figure out how the code coverage went DOWN with cross package testing enabled. 🙂 These numbers don't make much sense to me: https://github.com/vitessio/vitess/pull/14967#issuecomment-1919774260

But that's an unrelated issue. Thanks again!

[thomasrockhu-codecov]

@Andrew-S-Rosen.

This means that for most use cases, you will still need a token, even for public repositories.

This is true for all use cases except for PRs from forks to upstream repos. Sorry that this is a bit confusing, you're right.

The suggestion is to add the token as you have written out. PRs submitted from forks should have their code coverage reported as is.

[Andrew-S-Rosen]

@thomasrockhu-codecov --- alright, thank you for the clarification. I'll merge the change and see what happens when external contributors start opening PRs. Fingers crossed.

[mrazauskas]

Can it be that usage example in the blog post and current documentation of the codecov-action are incorrect?

They suggest passing token like this:

- name: Upload coverage reports to Codecov
- uses: codecov/codecov-action@v4
  env:
    token: ${{ secrets.CODECOV_TOKEN }}

That fails for me. But the following worked:

  - name: Upload coverage reports to Codecov
  - uses: codecov/codecov-action@v4
-   env:
+   with:
      token: ${{ secrets.CODECOV_TOKEN }}

---

With this change I see: "Process Upload complete" in the action log. Otherwise I get an error.

[thomasrockhu-codecov]

@mrazauskas ~~that is unexpected, it should work with env (as well as with). Can you share the logs in the failure scenario?~~

I see my error, it should have been

env:
  CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

I will update this now

[ljharb]

I use codecov on nearly 500 repositories, spread across many orgs and users. Some of the repos I don't have admin on (they're user-owned repos that I maintain but don't control). I use this via a shared action, so I can make one change and affect every repo at once.

It kind of seems like this means i will literally never be able to upgrade to v4, since even if I was capable if installing a secret on every repo I need to manage, I would never want to (because of the quantity).

Is this accurate? If so, what are the plans to either never drop support for v3, or, fix v4 so this isn't a requirement?

(given that v3 uses node16, which is deprecated, the former may not be an option)

(cc @chadwhitacre for visibility)

[thomasrockhu-codecov]

@ljharb, I realize that that is a huge pain for you. The closest thing I can think of that is offered in v4 is the global upload token, which is set per organization. You can find it at a similar URL: https://app.codecov.io/account/gh/orgname/org-upload-token

I think for your use case, you could add that token at the organization level as CODECOV_ORG_TOKEN to each organization (as opposed to each repository). Then, the shared action could intake

env:
  CODECOV_TOKEN: ${{ secrets.CODECOV_ORG_TOKEN }}

It's obviously not perfect, but hopefully will allow you to continue to use the shared action. As of now, there are no plans to drop support for v3.

[ljharb]

Again, i have many repos that are owned by another user, which means i can’t set a secret at any level. Anything that requires repo configuration is a nonstarter for me.