Github Actions raises a 422 error

[TheKevJames]

This is a known issue -- there are some cases where Github actions needs to be run with the github-actions service name and some cases where it must use the github service name. We have as of yet been unable to determine what causes the difference in usecase.

Running coveralls on Github Actions will use the github-actions service name by default. If you see a 422 error get raised, please try running coveralls --service=github, instead:

coveralls --service=github

If anyone discovers the secret sauce behind when coveralls requires each name, please let me know -- I'd love to fix the auto-detection!

[vchrombie]

Hi @TheKevJames, thanks for your work with this tool.

I was following up on the recent updates to this tool. I came across this workaround and I was trying to use this.

I was trying to migrate a few repositories from travis to github actions and I faced the 422 error even after using the coveralls --service=github. Strange is that it worked in one of the similar repos earlier but not working in this.

:+1: grimoirelab-perceval-opnfv build and its workflow file :-1: grimoirelab-perceval-mozilla build and its workflow file

[andy-maier]

Just in case the additional information helps: In our projects, the 422 error only shows up on Python 3.9 with latest package levels. Not on Python 3.9 with our minimum package levels, and not on older Python versions. See for example this Actions run: https://github.com/pywbem/pywbem/actions/runs/484671222

[liskin]

@TheKevJames Do we have some examples of situations where github-actions service name is needed? Most of the commits linked to this issue are switching back to github, and that is what we'll need to do in taskwiki as well. The tests there use the GITHUB_TOKEN provided by Actions, no COVERALLS_REPO_TOKEN, and there is COVERALLS_PARALLEL=true. With github-actions service name, we get 422.

I was involved in https://github.com/TheKevJames/coveralls-python/pull/227/ which made it possible to do test matrix builds with coveralls correctly, so I might be able to remember something, but my memory of it is somewhat foggy.

(I also think that service_job_id only needs to be specified with github-actions service name. With github, it's worked just fine without it, as it should: GH Actions don't provide any unique GITHUB_JOB_ID. It is probably harmless to pass null, though.)

[liskin]

@andy-maier

Just in case the additional information helps: In our projects, the 422 error only shows up on Python 3.9 with latest package levels. Not on Python 3.9 with our minimum package levels, and not on older Python versions. See for example this Actions run: https://github.com/pywbem/pywbem/actions/runs/484671222

That's because python 3.9 + latest package levels is the only configuration that actually uses the new coveralls-python 3.0.0. All the other use 2.1.2, which isn't broken by f4faa92dd8c9e9a523a6e778f36b13496ad79760 yet.

[liskin]

@nickmerwin Would you perhaps be able to shed some light on what's the difference between github and github-actions service names from the coveralls API point of view? The documentation (https://docs.coveralls.io/supported-ci-services and https://docs.coveralls.io/api-reference) is very hand-wavy on this: “This can be anything, but certain services have special features (travis-ci, travis-pro, or coveralls-ruby).” :-/

[TimoRoth]

My understanding of it is that if you are using the Github Actions natively provided GITHUB_TOKEN secret or ${{ github.token }}, you use the github service. Coveralls then works some internal magic, querying build and authentication info via GitHub APIs using that token. If you manually provide your coveralls token via a secret, you use github-actions (or really anything that's not a special-cased service name).

[Ri-Dearg]

Hi, thank you so much for this great package. I recently found myself facing this issue after updating to 3.0.0. I set a natively provided GITHUB_TOKEN secret. I didn't set any --service argument while using Github Actions in version coveralls 2.2.0 and it used to work just fine without one. After the upgrade to 3.0.0, I am facing the error 422 with no argument. Now I must manually set --service=github to successfully deploy the report. --service=github-actions fails. This would imply that @TimoRoth is correct. I don't know if that's any help to you in figuring this out. I ran a few tests to confirm, if you want to see the history, take a look here.

[tomato42]

from my own testing I can confirm what @TimoRoth said, if you use github service name (either through autodetection and by specifying it explicitly) you should use the github provided GITHUB_TOKEN secret, if you use anything else, including github-actions, you need to use the the token provided in the coveralls.io page

I can make both configurations work as long as I use the tokens and service names consistently.

[TimoRoth]

Are there really repositories that supply a variable called GITHUB_TOKEN, and put the Coveralls Token in there? Pretty sure in my original PR, bringing up proper support for GitHub Actions, I just assumed that GITHUB_TOKEN would be said Actions-Provided secret, and if it's not set, use the github-actions service instead of GitHub. Was there an issue with that approach?

If some people really do that, it should be possible to determine the kind of token from the length, even though that's a bit hacky. GitHubs tokens all appear to be 40 characters long, while the Coveralls tokens seem to be 32~33 characters long.

[vchrombie]

Hi, I have been watching this issue for more comments after https://github.com/TheKevJames/coveralls-python/issues/252#issuecomment-759968114, but everything worked fine until today.

We faced a new issue today, even after having the flag --service=github in the workflow. The build failed because of 422 error while coveralls reporting. It feels so strange because the build passed in my personal fork, from where the PR is sent, with the same commit.

I have added the -v flag for more clear logs. PR build log personal fork build log

I observed that there are two logs for the failed build. I have compared both the logs with the successful build log (I see differences only towards the end of the comparison file). successful and first attempt: https://www.diffchecker.com/OsIDZlEq successful and second attempt: https://www.diffchecker.com/jZuLQW2Q

I am really not sure what is the exact reason for this issue. I'm looking for some help with this.

On a side note, I see that the tool does a second attempt for reporting the coverage, if it fails because of 422 error, by generating a new job_id (wear function in api.py). But, GitHub Actions needs this to be null too, here it is "service_job_id":"None-9085385333557336431",.

I was thinking if this commit would be a correct fix for this problem.

[TheKevJames]

@TimoRoth yes, unfortunately we received several bug reports from folks who had done things the other way around, which led to the original work in trying to guess which service name was correct. You might be onto something with the length check, I think that's about the only potentially deterministic way of solving this I've seen so far.

Aside: I've just released @vchrombie 's fix as v3.0.1, so the Github retries should now work, at least!

[zgoda]

In my case the pattern is like:

• github fails for jobs running after regular push but succeeds eg for PRs after rebase
• github-actions exactly opposite

[afinetooth]

@liskin @TheKevJames, re:

@nickmerwin Would you perhaps be able to shed some light on what's the difference between github and github-actions service names from the coveralls API point of view?

Hoping to provide that input from the Coveralls side:

Note: I came to this issue through @zgoda's comment on this coveralls issue, yesterday:

---

The Coveralls API---specifically the /jobs endpoint, described in our API Reference here, expects that, when the incoming service_name parameter is github, it is receiving a job processed by the Coveralls Github Action.

As part of that, it expects a non-null value for service_job_id and a value for repo_token equal to ${{ secrets.GITHUB_TOKEN }} for the given repo. (which it expects to receive in the custom workflow env var, github-token, as described here.)

All together:

# incoming JSON params

"service_name": "github"
"service_job_id": <Some non-null value; the Coveralls Github Action sends $GITHUB_RUN_ID>
"repo_token": <Value of ${{ secrets.GITHUB_TOKEN }}>

On the other hand, the Coveralls API does not have a specific way of handling incoming jobs with service_name: github-actions. It treats these jobs the same way it treats jobs coming from a generic CI, which means that it expects repo_token to equal the <coveralls repo token> (provided by Coveralls, on the Coveralls Repo Page), and can accept null for service_job_id.

All together

# incoming JSON params

"service_name": "github-actions"
"service_job_id": <Anything, including NULL>
"repo_token": <Value of <coveralls repo token> provided by Coveralls; same as COVERALLS_REPO_TOKEN>

---

What this means for @zgoda's issue, is that, to avoid the error received in Step 5 of his reproduction steps, he can set up his project like this:

# .coveralls.yml
service_name: github-actions

And:

# workflow.yml
env:
    - name: Coveralls report
      if: matrix.python-version == 3.8
      env:
        COVERALLS_REPO_TOKEN: ${{ secrets.COVERALLS_REPO_TOKEN }}
        GITHUB_TOKEN: $COVERALLS_REPO_TOKEN
      run: |
        python -m pip install -U coveralls
        coveralls -v --service=github-actions

A little redundant / paranoid passing service_name twice in two different places, but probably safer. Similarly, passing COVERALLS_REPO_TOKEN in the workflow file is probably not necessary, but it at least makes it clear to users following Coveralls instructions for setting up a generic CI. Only GITHUB_TOKEN: ${{ secrets.COVERALLS_REPO_TOKEN }} should be required.

---

What this means for coveralls-python is that it can avoid the error in @zgoda's Step 5, encountered by some users, by changing the "resubmit post" code here.

Based on current code here, if the service_job_id parameter will be None/null, then the service_name parameter will have to be github-actions and the repo_token parameter will have to be the <coveralls repo token> (again, provided by Coveralls, on the project's Coveralls Repo Page). This value is the same as what Coveralls would have you pass as COVERALLS_REPO_TOKEN, and means that, in the coveralls-python context, the GITHUB_TOKEN env var in the workflow file should be set to the <coveralls repo token>, as opposed to the ${{ secrets.GITHUB_TOKEN }}, as currently prescribed here.

All together:

# incoming JSON params

"service_name": "github-actions"
"service_job_id": <Anything, including None/NULL>
"repo_token": <Value of <coveralls repo token> provided by Coveralls; same as COVERALLS_REPO_TOKEN>

The difficulty here, of course, is that your users may not have provided their <coveralls repo token>, which only they can source form their Coveralls Repo Page. In that case, an alternate:

Alternately, since there is a conditional here that can result in service_job_id being non-NULL, in this case, service_name could be set to github, in which case, repo_token would have to be set to ${{ secrets.GITHUB_TOKEN }}. Which is something you might need to massage into reality for the user, given how they have their project currently set up.

All together:

# incoming JSON params

"service_name": "github"
"service_job_id": <Some non-null value; the Coveralls Github Action sends $GITHUB_RUN_ID>
"repo_token": <Value of ${{ secrets.GITHUB_TOKEN }}>

But we'd probably recommend keeping it simple and sticking with service_name: github-actions (everywhere) and requiring repo_token: <coveralls repo token>, vis-a-vis env var GITHUB_TOKEN in workflow file: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}. (As per the solution recommended here.)

This lets service_job_id be anything, including None/NULL.

---

I believe this should resolve issues related to needing to pass different values for service_name, etc. But if you make appropriate changes and your users continue reporting "mileage may vary" experiences, let us know and we'll look more closely.

[liskin]

@afinetooth Are you sure about service_job_id having to be non-null with service_name being github? My experience differs: we've been submitting runs with service_name=github, null service_job_id (as mentioned previously, GitHub doesn't provide any unique job id at the moment, so there's no reliable way to get this) and non-null service_number (to connect the parallel jobs together), and it's been working fine. Is that perhaps a parallel=true thing?

Also, your recommendation to use service_name: github-actions with the coveralls repo token won't work with pull requests, because action runs for pull requests don't have access to secrets. We need a way to make this work with GITHUB_TOKEN.

[afinetooth]

@liskin, looking at it more closely, you're right, the job can get through with null for service_job_id. The reason I described it as "required" is because the Coveralls Github Action controls/ensures that input, and because we have a place in our UI code that checks whether the job came via Github Actions, with a method .github? defined as follows:

  def github?
    self.service_job_id.present? && self.service_name == "github"
  end

The method using github? constructs the CI URL back to the build / workflow run on Github Actions.

TBH, I'm not sure whether the "generic" route results in the same, correct CI URL.

Perhaps you can verify for me by checking one of your coveralls builds.

On your coveralls build page, the JOBS section would look like this, with an active link back to the correct workflow run:

Do you see that?

If not and you can live without it, there's no need to change your setup.

[liskin]

@afinetooth I can check, yeah. Here for example: https://coveralls.io/builds/38506836, and there is no link, even though we're submitting with service_name=github.

And again, there cannot be any link, because GitHub doesn't provide enough info to generate that link for parallel jobs. The "713375952" number is correct, but it refers to the entire workflow run https://github.com/tools-life/taskwiki/actions/runs/713375952. The individual job runs have links like this: https://github.com/tools-life/taskwiki/runs/2257937287, https://github.com/tools-life/taskwiki/runs/2257937296, … and these "2257937287", "2257937296" numbers (the only ids that are actually unique enough for passing to service_job_id) simply aren't available anywhere from within the run environment.

So yeah, I would love to have that active link that connected coverage results of individual parallel job runs, but since GitHub doesn't provide the info to create that link, I must do without it. :-/

I mentioned this several times to GH Actions devs: https://github.com/actions/toolkit/issues/65#issuecomment-654220494, https://github.community/t/add-build-number/16149/17 but so far they've ignored it. If anyone else wants to see coveralls link back to invidual job runs, can you please express your support for GitHub exporting GITHUB_JOB_ID by commenting in the linked threads?

---

Oh and speaking of the Coveralls Github Action, I think it's implemented incorrectly. It sets a non-unique value to service_job_id, which obviously breaks parallel runs. The suggested workaround is to manually set the flag-name parameter to disambiguate the runs, but that breaks as soon as any run is reexecuted, as then not even the service_job_id and flag-name pair is unique any more. Setting neither service_job_id nor flag-name works a bit better.

It might be nice if you changed the backend to generate a workflow-wide link from service_number as well, as that's where the non-unique workflow id goes (I mean, that's where it should go, and that's where coveralls-python puts it). If/when GitHub exposes the the individual job id, we can start submitting that id as service_job_id and then linking to individual parallel runs will work as well, but having a link to the entire workflow somewhere would be nice in the meantime.

[afinetooth]

@liskin thanks for the additional input here. Doing some local testing by forking the repo of my o.p., @zgoda, and may need some more examples, of which yours and @vchrombie's are good options.

[afinetooth]

FYI, @TheKevJames, @liskin, I completed tests of my own fork of @zgoda's repo following steps he described that continually reproduce the 422 error for him.

As you can see from my comment here, I was unable to reproduce the error.

I have asked for @zgoda's review to make sure I reproduced the steps properly, but the results were as I expected based on the behavior I describe above for jobs with service_name: github-actions vs service_name: github.

Per my recommendation, I configured my workflow file to ensure the success conditions for jobs with service_name: github-actions.

Which is primarily making sure GITHUB_TOKEN = COVERALLS_REPO_TOKEN (and not secrets.GITHUB_TOKEN):

# .github/workflows/test.yml

[...]

    - name: Coveralls report
      if: matrix.python-version == 3.8
      env:
        COVERALLS_HOST: https://3d06fff8dca5.ngrok.io
        COVERALLS_REPO_TOKEN: ${{ secrets.COVERALLS_REPO_TOKEN }}
        GITHUB_TOKEN: $COVERALLS_REPO_TOKEN
      run: |
        python -m pip install -U PyYAML coveralls
        coveralls -v --service=github-actions

COVERALLS_HOST above is the tunnel to my dev coveralls.

Would anyone else like to try these workflow settings to see if it helps them avoid the 422?

Will it help if I test someone else's repo?

---

P.S. @liskin, I haven't addressed the issue you raised about how Coveralls Github Action is implemented, for the inclusion/exclusion of links back to workflows. Will need to address that elsewhere. For now, just know that service_name: github-actions is processed the same as a generic CI, which results in no job links back to CI.

[TimoRoth]

I still think it's odd to put anything but the GITHUB_TOKEN into the env var called GITHUB_TOKEN. Putting the coveralls provided token in there seems very confusing for both users and obviously the code.

Was it ever documented to be like that?

[liskin]

Not just odd, using a token like COVERALLS_REPO_TOKEN (regardless of what env var it goes into) doesn't work in pull requests, as GitHub doesn't expose secrets in pull requests (for obvious reasons). Recommending people to do that is irresponsible.

[afinetooth]

I still think it's odd to put anything but the GITHUB_TOKEN into the env var called GITHUB_TOKEN. Putting the coveralls provided token in there seems very confusing for both users and obviously the code. Was it ever documented to be like that?

@TimoRoth I tend to agree but afaik GITHUB_TOKEN was introduced by coveralls-python, and needed to be used in my example; but that original variable name makes sense since the creator expected to be passing ${{ secrets.GITHUB_TOKEN }}.

Not just odd, using a token like COVERALLS_REPO_TOKEN (regardless of what env var it goes into) doesn't work in pull requests, as GitHub doesn't expose secrets in pull requests (for obvious reasons). Recommending people to do that is irresponsible.

@liskin I wasn't aware of that, though I see you mentioned it previously. Is that just a recommendation from Github? or will Github Actions simply not surface secrets in the context of a PR? My expectation would be that it wouldn't surface those for forked PRs (that makes sense to me).

This also seems to go against my experience with this PR: https://github.com/afinetooth/flask-rollup/pull/1

Where, after changing all token references to COVERALLS_REPO_TOKEN, this Action run succeeded for the pull_request build: https://github.com/afinetooth/flask-rollup/runs/2452128424

Please note, I'm not saying Coveralls recommends using COVERALLS_REPO_TOKEN. In fact, we don't. Our implementation of the Coveralls Github Action uses ${{ secrets.GITHUB_TOKEN }}. The issue is just that any incoming CI job with a service_name other than github will be treated like a job from a generic CI, which expects COVERALLS_REPO_TOKEN.

@TimoRoth @liskin @TheKevJames please stand by. I incorrectly reproduced the use case from @zgoda so I will need to re-try with his correction here to re-verify my assumptions.

[liskin]

@liskin I wasn't aware of that, though I see you mentioned it previously. Is that just a recommendation from Github? or with Github Actions simply not surface secrets in the context of a PR? My expectation would be that it wouldn't surface those for forked PRs (that makes sense to me).

Oh, perhaps it does expose secrets when the PR is from the same repo. Mea culpa.

It definitely does not when the PR is from a fork, for security reasons. So if you want to support pull requests in general, and I hope you do, then using COVERALLS_REPO_TOKEN is not an option.

[afinetooth]

Oh, perhaps it does expose secrets when the PR is from the same repo. Mea culpa.

All good! I'm having trouble finding a decisive answer on it. It seems secrets.GITHUB_TOKEN may be read-only for forks.

It definitely does not when the PR is from a fork, for security reasons. So if you want to support pull requests in general, and I hope you do, then using COVERALLS_REPO_TOKEN is not an option.

While it worked for my local PR, this resource indicates you're right about custom env vars.

However, like I said, Coveralls does not recommend using COVERALLS_REPO_TOKEN when using Github Actions. We recommend using ${{ secrets.GITHUB_TOKEN }} when using service_name: github.

My recommendation was a limited one, to avoid the intermittent 422 error experienced by coveralls-python users when rebasing a PR as described here; and that is in lieu of changes to coveralls-python , which is why I was posting here (for visibility).

[TimoRoth]

In my original PR adding support for the actual 'github' service type, I also introduced use of the env variable called GITHUB_TOKEN: https://github.com/TheKevJames/coveralls-python/commit/f597109b62fadaf900af79d4f08a7debee5229e2

Before that, I don't think it was used or supported at all. I'm not sure at which point it got its double role to also get the coveralls repo token passed in, but the original idea there was that GITHUB_TOKEN is always expected to have the actual GITHUB_TOKEN, and not something else. If you for some reason need/want to pass in the actual coveralls token, there's a separate env var for that. So the change of the default service from 'github' to 'github-actions' is still confusing me.

[afinetooth]

Hi @TimoRoth, I guess I introduced the double-usage of GITHUB_TOKEN in my (limited) example to (ideally, temporarily) work around the 422 error described in this issue. As I explain above, the change to the value passed for GITHUB_TOKEN is required to avoid the error due to the context I described around the way the Coveralls API handles incoming jobs with service_name: github vs service_name: github-actions.

I felt it was something my example needed to address because of the first error message my OP described receiving---from this line:

Running on Github Actions but GITHUB_TOKEN is not set.
Add "env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}" to your step config.

Since that check pre-existed in the coveralls-python code, I figured I needed to work with that env var as-is.

Since I knew the error message was asking the user to set GITHUB_TOKEN but that the Coveralls API would be expecting COVERALLS_REPO_TOKEN (if service_name: github-actions), I suggested setting GITHUB_TOKEN equal to COVERALLS_REPO_TOKEN.

Ideally, users would only set one repo_token-related env var. Which they can if they settle on one value for service_name.

Again, I posted to this issue with the background info requested here so that coveralls-python creators could modify the codebase as they see fit with that info.

To clarify the service_name distinction further:

• service_name: github will be handled as though the job is coming from Github Actions, but packaged up the way the Coveralls Github Action packages up jobs. This source file shows how the Github Actions env vars are handled by the Coveralls Github Action. In particular, this line shows that Coveralls Github Action sets Coveralls' standard COVERALLS_REPO_TOKEN var to ${{ secrets.GITHUB_TOKEN }} through a CI env var it expects to receive, github-token, as described here in the README. FWIW, the Coveralls Github Action also sets a value for service_job_id, for which it uses GITHUB_RUN_ID per this line. I guess if there's any point I'm ultimately trying to make here, it's that, for any integration that wants to leverage Github Actions for CI, it would be safest to use this code from the Coveralls Github Action as a spec for sending jobs to the Coveralls API.

• service_name: github-actions will be treated as a job coming from a generic CI, same as service_name: anything_at_all, a case in which the Coveralls API will be expecting its standard COVERALLS_REPO_TOKEN to equal the repo token Coveralls assigns all repos, to be found in the repo's start page or settings page.

Hope that clarifies.

I'm sure some people are successfully submitting service_name: github with COVERALLS_REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}, but all I'm saying is that to continue doing so without being aware of the above is to risk a "mileage may vary" situation like the one described here in the coveralls-python README.

[liskin]

• FWIW, the Coveralls Github Action also sets a value for service_job_id, for which it uses GITHUB_RUN_ID per this line.

Can we please stop suggesting that people put the non-unique GITHUB_RUN_ID into the required-to-be-unique service_job_id field? It only kind of maybe works for non-parallel jobs and breaks horribly for parallel jobs. GITHUB_RUN_ID can only go into service_number.

I guess if there's any point I'm ultimately trying to make here, it's that, for any integration that wants to leverage Github Actions for CI, it would be safest to use this code from the Coveralls Github Action as a spec for sending jobs to the Coveralls API.

Maybe, just maybe, the better way forward would be to actually fix the Coveralls Github Action?

Thanks!

[afinetooth]

I'm ok with that but, personally, I'm just a support engineer so my goal is to help users get their builds sending coverage to coveralls and provide background on the Coveralls API to integrators.

I'm happy to relay the request as a feature change, which I'll do now, and even argue for it.

(That said, I think we're on the same page that without a unique identifier from Github Actions, Coveralls won't work as designed for parallel jobs, but that this should be limited to incorrect links back to CI jobs on the builds page.)

Upon doing so, are we good for our purposes here?

Update: I have connected my request to your issue (https://github.com/coverallsapp/github-action/issues/59) @TimoRoth

[liskin]

Indeed, you understanding what the issue is, relaying it and vouching for it internally is the best possible outcome. So we're good! (Sorry for being very direct and possibly harsh. I let my frustration with this year-old issue bubble out. Sorry!)

[afinetooth]

I understand! it's been around for a long time and you guys have tried to work around it and a lack of documentation on it admirably. We're good. Hopefully we can put it to bed soon.

[Panaetius]

It'd be great if the docs could be updated with the difference between service_name: github and service_name: github-actions and how they relate to the coveralls and github tokens. Currently the documentation is quite incomplete in this regard.

[pbelskiy]

So, how should I use coveralls now for both PR and my commits? I got 422 now. Any final solution now?

Seems okay configuration now - https://github.com/pbelskiy/aiojenkins/blob/master/.github/workflows/python.yml

[stephane-caron]

In case it can help others, I got stuck on 422 with the following config:

            - name: "Coveralls"
              env:
                  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
              run: |
                  coveralls --service=github

Tried out other environment variables, setting the service name to both "github" and "github-actions", to no avail.

Then I disabled my repository on https://coveralls.io/ (deleting all its data) and re-enabled it manually. After that operation, the config above started working again.

[kennedykori]

@stephane-caron solution worked for me too.

[leeqvip]

I solved it this way, removing the repo from https://coveralls.io/ and adding it back.