https://docs.coveralls.io/parallel-build-webhook should document all of the webhook parameters completely

[exarkun]

https://docs.coveralls.io/parallel-build-webhook only gives partial documentation for the webhook which is not sufficient to understand how to use it except in a couple narrow configurations (TravisCI, CircleCI). If it documented all of the parameters and their meanings then anyone could fairly easily use the webhook from any CI system.

The first problem with the page is this statement:

Then set up a post build webhook like so: https://coveralls.io/webhook?repo_token=(your repo token)

This makes it seem as though no additional parameters are required. However, reading the rest of the page, it seems quite unlikely this is true.

The rest of the problem is that the webhook payload is not explained:

{
  "payload": {
       "build_num": 1234,
       "status": "done"
     }
}

How is the value for build_num assigned? What are the consequences of the value? What are the possible values of "status" and what are their consequences?

[afinetooth]

@exarkun, thanks for the feedback and suggestions.

We're in progress on a major update to our documentation.

I assume what you'd like to see is something like this: https://docs.coveralls.io/api-reference

Or this: https://coveralls.io/api/docs

(If it had included the /webhook endpoint.)

Here's an attempt, which, with your feedback, could function as a draft of our updated documentation:

(I'll be making updates to this as we proceed.)

---

WEBHOOK (V1)

CLOSE A PARALLEL BUILD

ENDPOINT

POST https://coveralls.io/webhook

PARAMETERS

NAME	TYPE	DESCRIPTION	EXAMPLES
repo_token	String	Coveralls repo token, provided in Coveralls repo settings. Required for private repos. Likely set in CI env var: COVERALLS_REPO_TOKEN. Exception: The Coveralls Github Action uses github-token ({{ secrets.GITHUB_TOKEN }}) in lieu of COVERALLS_REPO_TOKEN since the action has privileged access to the repo and this distinguishes it from other integrations using Github Actions for CI.	Ac9rqOK8Lnvq2ObBYFklb7ci2yUBd3Ytz
payload[build_num] (required)	String	Build number for build to be closed. Most likely, the same value passed as service_number with original job. Details here.	107, 7b491012-9a7c-4ac0-a620-04e5c0695da1
payload[status] (required)	String	Only a value of "done" will result in the build being closed.	done

REQUEST

ROUTE

/webhook

HEADERS

Accept: application/json
Content-Type: application/json

BODY

{
  "repo_token": "Ac9rqOK8Lnvq2ObBYFklb7ci2yUBd3Ytz"
  "payload": {
       "build_num": "7b491012-9a7c-4ac0-a620-04e5c0695da1",
       "status": "done"
     }
}

USAGE

This command format can be used for an example shell script or CI step:

curl -k $COVERALLS_ENDPOINT/webhook?repo_token=$COVERALLS_REPO_TOKEN -d "payload[build_num]=$BUILD_NUMBER&payload[status]=done"

Where:

VARIABLE	TYPE	DESCRIPTION	EXAMPLE
$COVERALLS_ENDPOINT	String	Coveralls API base URL. Likely set as CI env var.	https://coveralls.io
$COVERALLS_REPO_TOKEN	String	Coveralls repo token for private repos. Likely set as CI env var./p> Exception: The Coveralls Github Action uses github-token ({{ secrets.GITHUB_TOKEN }}) in lieu of COVERALLS_REPO_TOKEN.	Ac9rqOK8Lnvq2ObBYFklb7ci2yUBd3Ytz
$BUILD_NUMBER	String	Likely, value for service_number in original job POST to https://coveralls.io/api/v1/jobs.	7b491012-9a7c-4ac0-a620-04e5c0695da1

EXAMPLE

curl -k "https://coveralls.io/webhook?repo_token=Ac9rqOK8Lnvq2ObBYFklb7ci2yUBd3Ytz" -d "payload[build_num]=7b491012-9a7c-4ac0-a620-04e5c0695da1&payload[status]=done"

EXPECTED RESPONSES

When all values are correct:

• repo_token matches the repo_token of an existing repo
• payload[build_num] matches the service_number of a build for the same repo

{"done":true,"url":"https://coveralls.io/builds/35976056","jobs":2}

When payload[build_num] is incorrect:

{"error":"No build matching CI build number 7B491012-9A7C-4AC0-A620-04E5C0695DA1 found"}

When repo_token is incorrect:

{"error":"Invalid repo token"}

When repo[status] is not set to "done":

{"error":"Status is not 'done'"}

---

[exarkun]

Thanks @afinetooth, that looks really good. In particular, the explanation that payload[build_num] needs to match service_number of a build (one or more builds?) is the key piece that I spent a lot of time today trying to figure out.

One step further on this might be to make build_num a backwards compat-only parameter and instead prefer payload[service_number] - using the shared name to more strongly hint at the connection here.

But even the documentation as written in your comment would probably have quickly gotten me where I needed to be.

[afinetooth]

@exarkun I see. To answer your question, for all intents and purposes, consider service_number unique to a single build (to which there may belong one or more jobs). Technically, only a build's internal ID is unique, and service_number can be re-used, such as when a user re-runs a CI build that re-POSTs coverage to our API. In those cases, we focus on the latest build.

Strongly agree re: payload[service_number], great suggestion. That said, I wonder if changing the original service_number to build_number and payload[build_num] to payload[build_number] might not provide even more clarity. At the very least, it'd be good to allow for aliases that match, so I've submitted the change request.

Helpful to know we're in the ballpark. Thanks.

[RoadRunnr]

The description for repo_token is wrong when being use from a GitHub Action.

While digging through the official GH Action for coveralls, I found that the repo_token is set to secrets.GITHUB_TOKEN. This is not the token from coveralls, but a token generated by GH. It seems that coveralls has implemented something special for GH Action to make it work with that token.

[afinetooth]

@RoadRunnr this is true. Only in the case of Github Actions, where we have privileged access to your CI service because Coveralls is already integrated with Github, we use Github's identifier for the repo:

github-token: ${{ secrets.GITHUB_TOKEN }}

In fact, we don't even ask for repo_token, but under the hood the Coveralls Github Action passes the github-token as the repo_token. In fact, any Github Actions workflow can use:

github-token: ${{ secrets.GITHUB_TOKEN }}

Since we recognize the CI POST is coming from Github.

Insofar as the API Reference is a general doc about the API, what if we added this note to the existing params definition?:

PARAMETERS

NAME	TYPE	DESCRIPTION	EXAMPLES
repo_token	String	Coveralls repo token, provided in Coveralls repo settings. Required for private repos, except when using Github Actions for CI (details). For other CI services repo_token can be set with the env var: COVERALLS_REPO_TOKEN.	Ac9rqOK8Lnvq2ObBYFklb7ci2yUBd3Ytz
payload[build_num] (required)	String	Build number for build to be closed. Most likely, the same value passed as service_number with original job. Details here.	107, 7b491012-9a7c-4ac0-a620-04e5c0695da1
payload[status] (required)	String	Only a value of "done" will result in the build being closed.	done

[Panaetius]

In the case of using coveralls for a python based repo, we don't really want to send a Github token, if COVERALLS_REPO_TOKEN would suffice. Sending a Github Token to a third party has certain security implications and is not something that you necessarily want to publish in any way.

I find the behavior of this endpoint really odd. COVERALLS_REPO_TOKEN is the token from your service to communicate with your service, why should a Github token (Specific to one potential type of of service) take precedence over the coveralls token, which is more generic. Especially if there is no way to use the coveralls token instead of the github token at the moment, when running inside github actions.

I just spend like 6 hours trying to figure out a problem with this until I came across this Github issue. It's quite unexpected behavior. Especially having an endpoint behave differently, depending on where it is called from, with exactly the same URL and payload, is weird to me. Maybe you could change it to be more explicit, with a github_repo_token field instead of reusing repo_token.

Also it'd be nice if this could be added to your docs proper, so it's easier to find. It's been a while since this issue was opened.

[afinetooth]

@Panaetius using the GITHUB_TOKEN is a means of identifying the repo to the Coveralls API when using the Coveralls Github Action and distinguishing it from other integrations not using the Coveralls Github Action, but still using Github Actions for CI.

Integrations using Github Actions for CI are welcome and encouraged to use COVERALLS_REPO_TOKEN to identify their repos, as long as they use a different service_name param from that used by the Coveralls Github Action, which is github. We usually recommend github-actions, but any value will do as the Coveralls API handles these posts like those from any other CI service.

According to Github's Authentication in a workflow, it is normal and safe to use the GITHUB_TOKEN to authenticate on behalf of the GitHub App installed on your repo by enabling a Github Action, particularly since the ability to reference it in a workflow file with the syntax ${{ secrets.GITHUB_TOKEN }} redacts it from build logs and maintains the permissions profile that you set for it.

This was the recommended practice when we built our official GH Action and, as I say above, left a means of distinguishing it from other integrations using Github Actions.

Point taken on the specific/generic usage. In this case, COVERALLS_REPO_TOKEN is the more generic use as being applicable to your Coveralls repo, while GITHUB-TOKEN is the more specific use as applying to one integration pathway to that repo.

[Panaetius]

Maybe to go a bit more into detail on what the issue was for us. We've been using the coveralls python package in our Github Actions.

There, the coveralls steps look like this:

- name: Coveralls
      env:
        COVERALLS_REPO_TOKEN: ${{ secrets.COVERALLS_REPO_TOKEN }}
        COVERALLS_PARALLEL: true
      run: coveralls

and the coveralls repo token works perfectly fine.

However, this doesn't work:

- name: Aggregate Coveralls
      env:
        COVERALLS_REPO_TOKEN: ${{ secrets.COVERALLS_REPO_TOKEN }}
      run: coveralls --finish

because it will treat the COVERALLS_REPO_TOKEN as a GITHUB_TOKEN. The only issue related to this that can be easily found on google is https://github.com/TheKevJames/coveralls-python/issues/248 which says to specify a GITHUB_TOKEN instead, however, setting the GITHUB_TOKEN doesn't work (Since it's not in the action, I guess). Not that we should have to set the github token. Looking at the source, it sets the service-name to github-actions when running inside GH Actions, which I think cannot work in this scenario.

The documentation at https://coveralls-python.readthedocs.io/en/latest/usage/configuration.html#github-actions-support is also quite confusing, stating that one should set service_name to github or github-actions, neither of which seems to work with either a COVERALLS_REPO_TOKEN or a GITHUB_TOKEN for the --finish action, at least when I tested it.

We ended up using the coveralls github action for the --finish as that was the only thing that works, we couldn't get it to work with the coveralls package. And we added

            permissions:
              contents: read

to restrict the permissions of the GITHUB_TOKEN used.

The expectation as a user was that coveralls --finish should just work with a COVERALLS_REPO_TOKEN just the same as the other coveralls commands.

[afinetooth]

@Panaetius I continually re-confuse myself when looking into similar issues, so, if you don't mind, please refer to this comment , where I attempt to clarify the distinctions at play when using coveralls-python via Github Actions vis-a-vis the service_name and repo_token parameters.

I understand you've solved your current issue, which is great. My hope is that, if desired, you might be able to use the information there to revert either the use of GITHUB_TOKEN, or the use of two separate integrations, coveralls-python & Coveralls Github Action.

If not, and you still have questions, let me know here and I'll try to help.

[Panaetius]

Thanks for the information, that helps clear things up! I commented on that issue.

[stale[bot]]

This issue has been automatically marked for closure because it has not had recent activity. It will be closed if no further activity occurs. If your issue is still active please add a comment and we’ll review as soon as we can. Thank you for your contributions.