Coveralls doesn't block PR if test coverage decreased

[st4lk]

When I'm creating new PR with decreased test coverage - coveralls doesn't block the PR although test coverage decreased (comparing to source branch). It shows that my branch in PR has "New Coverage". If I'll push new commit to the branch associated with PR - only after that PR will be blocked by coveralls because test coverage decreased. I'm wondering: why coveralls doesn't block the PR if coverage decreased on first commit?

Stack of tools I'm using

• CircleCI to make the build
• coveralls-python to push coverage report to coveralls from CircleCI
• I've enabled all the threshold alerts in coveralls settings:
• This is open source repository in github

Maybe I need to adjust some settings in coveralls.io somewhere?

[st4lk]

Looks like can be the same issue as described here: https://github.com/lemurheavy/coveralls-public/issues/1167

[afinetooth]

Hi, @st4lk. So I hear what you're saying as far as the description of your issue, however some details are throwing me off.

First, from your screenshot above, you can see under BUILD TYPE (below the read header reading "FIRST BUILD ON...") that your build type is push, and not pull_request. Therefore, any settings related to pull requests would not apply for that build. In other words, Coveralls would not send a failing status update to Github, blocking the PR from being merged (because it's not a PR).

I believe I have identified the build you're using as your example, and it is the only build in your project with a creation date of 18 Mar 2022 10:52AM UTC, so I believe we're looking at the same one. (It's about 2/3 way down the page here in your build history.)

When I look further up the history toward more recent builds and get to the first build that is a PR---a build starting with 2C8E5BAF- created at 18 Mar 2022 12:43PM UTC---I see that---as a pull_request---that build also had a decrease in coverage:

And that a failing status update was sent for the PR.

From our database:

id=16719988705 description="Coverage decreased (-3.03%) to 63.636%" state="failure"

And from the PR:

[st4lk]

Hey @afinetooth , thank you for the quick response!

I think this is what is happening:

• new branch is pushed to github
• circleci start the build
• PR is not opened yet
• circleci have run all the tests and have pushed coverage report to coveralls
• since PR is not opened - coveralls don't block it
• finally, PR is opened, but it will not be blocked by coveralls even if test coverage felt down

It looks like a common workflow, so I'm wondering what would be the best thing to do. Obviously, there is some delay between pushing new branch to github and opening the PR for it.

Should we trigger coveralls once again when PR is actually opened so it can possibly block it?

(another approach could be to start the build by circleci only when PR is opened and not when branch is pushed, but it has some other disadvantages)

[afinetooth]

Hi, @st4lk.

• finally, PR is opened, but it will not be blocked by coveralls even if test coverage felt down

That's actually not true. Coveralls will send failing (blocking) status updates for all PRs when coverage falls below the thresholds in your settings.

It actually did that when the PR for your branch was opened.

Here's the PR build: https://coveralls.io/builds/47490484

And here's the blocking status update on the PR at Github:

Question: Did you configure Github to block PRs with failing status updates?

It looks like a common workflow, so I'm wondering what would be the best thing to do. Obviously, there is some delay between pushing new branch to github and opening the PR for it.

Should we trigger coveralls once again when PR is actually opened so it can possibly block it?

Yes, you should configure CircleCI to build pull_requests and, if you're using a Coveralls integration, Coveralls will do the rest, in terms of sending failing (blocking) status updates to Github.

Unless I misunderstand your workflow, Coveralls is already designed to support your workflow. Let me explain how:

• Coveralls will generate coverage reports for push builds and send status updates to Github for them. For example, here's the (failing) status update from Coveralls for the push build described above:

• When a pull_request is opened, Coveralls processes the PR build and also sends status updates, but, because the build is a PR at Github, the status update has the ability to block the PR from being merged (if it's failing).

I guess I'm confused about what the proper Coveralls feature would be to handle your workflow, since Coveralls can't know a new branch is intended to become a pull_request.

I think I understand why a failing status update would be useful for a new feature branch---so developer(s) know they won't have a mergeable PR until meet coverage requirements---however, as I say above, that feature is already there as far as Coveralls is concerned, because it's sending the failing status update.

Hope that makes sense. Let me know if I'm missing something.

[DysphoricUnicorn]

A similar thing happened for us just now: This build did not block the PR, even though it shows decreased coverage and is of type pull request.

Is there some setting I have to change to set the threshold to 100%?