Checks are failing even thought they are informational

[nwalters512]

Describe the bug

I've configured the project and patch status checks to be informational:

coverage:
  status:
    project:
      informational: true
    patch:
      informational: true

The documentation on https://docs.codecov.com/docs/commit-status states that the status checks should never fail when marked as informational:

Use Codecov in informational mode. Default is false. If true is specified the resulting status will pass no matter what the coverage is or what other settings are specified. Informational mode is great to use if you want to expose codecov information to other developers in your pull request without necessarily gating PRs on that information.

However, they're both failing on this PR: https://github.com/PrairieLearn/PrairieLearn/pull/9743

Environment:

• Browser: N/A
• System: CI (GitHub Actions)

To Reproduce

I'm not entirely sure what to put here; I think I've given as much information as I can. A repo containing the above YAML configuration does not behave as the documentation says it should. This is visible on https://github.com/PrairieLearn/PrairieLearn/pull/9743.

Expected behavior

Per the documentation, informational checks should never fail.

Screenshots

[drazisil-codecov]

Hi @nwalters512 ,

You are missing the check's name.

coverage:
  status:
    project:
      # Don't fail the status check if coverage decreases.
      informational: true # I am the name of the check.
    patch:
      # Don't fail the status check if coverage decreases.
      informational: true

Should be

coverage:
  status:
    project:
      # Don't fail the status check if coverage decreases.
      default: # This is the check name. It doesn't have have to be default , it just needs to exist.
        informational: true
    patch:
      # Don't fail the status check if coverage decreases.
      default: 
        informational: true

[drazisil-codecov]

@rohan-at-sentry I called this out here https://docs.codecov.com/docs/codecovyml-reference#coveragestatus, but it appears to have never made it to (or was lost from) the main docs.

[nwalters512]

Thanks for pointing that out; sorry I missed that in the docs! For what it's worth, I ran my config through the validator as recommended in your docs and it said it was valid:

curl --data-binary @.codecov.yml https://codecov.io/validate
Valid!

{
  "comment": {
    "require_changes": true,
    "require_base": true,
    "require_head": true
  },
  "coverage": {
    "status": {
      "project": {
        "informational": true
      },
      "patch": {
        "informational": true
      }
    }
  }
}

It would be very helpful if the validator complained about my config. On a closer look, I see that the docs at https://docs.codecov.com/docs/commit-status#disabling-a-status state the following:

Note that you can not enable a status by setting it to yes, you need to define it by using the structure described here

It seems as though neither the validator nor the implementation agree with this.

[drazisil-codecov]

It seems as though neither the validator nor the implementation agree with this.

:-/ 👀

[drazisil-codecov]

It's because in order to handle "False" it's of type "boolean"

https://github.com/codecov/shared/blob/f17719ac12d8e8bf60c7b51af5716b001ccdff0e/shared/validation/user_schema.py#L302

So it's "valid", just...doesn't do anything 🤐

@rohan-at-sentry I feel there's room for improvement here, but I don't know the safest way to modify something so core to Codecov operations. I added the note, but It's a valid point that if it validates....