inconsistencies with CodeCov rejections

[jywarren]

Possibly in relation to #747, I'm seeing that some PRs are blocked if they have no tests, which is fine, but that when adding tests,

1. we can't seem to get the Codecov reported coverage for the PR to change
2. CodeCov's info seems inconsistent; initially this PR (https://github.com/publiclab/mapknitter/pull/801) reflected a drop in coverage to the sessions controller,in CodeCov's comment. But after rebasing, now it shows a drop in the Annotations controller:

Why might CodeCov be varying in its reporting? cc @kaustubh-nair

Thank you!

[jywarren]

I'm afraid this is blocking Mapknitter 3 launch, so I'm going to try to turn down the strictness of CodeCov blocking PRs for coverage... at least for now!

But I don't think it's really working properly -- for example, https://github.com/publiclab/mapknitter/pull/811/files really won't drop coverage by almost 5% -- that can't be right!

[jywarren]

OK, so i turned patch off:

coverage:
  status:
    project: on
    patch: off

[jywarren]

https://github.com/jywarren/mapknitter/commit/9b92a9090d299d0715219638e77d538c4a590ac5

[jywarren]

OK, so for example, this doesn't make sense - we aren't substantially changing anything but an HTML fragment, but it's reporting a near 7% drop in coverage. I need to merge that PR to publish MapKnitter so I think i'll override with my admin privileges, because the 7% drop can't be correct.

https://codecov.io/gh/publiclab/mapknitter/compare/0d686d8073cd945b08ab7557ad93676be7ab58fc...e4268dc0efa0304b4fd57ab60706052674705610/diff

Thanks for any help!

[kaustubh-nair]

This is really weird.. It makes sense that coverage shouldn't drop by two percent for https://github.com/publiclab/mapknitter/pull/801 I generated the reports locally to confirm this.

[kaustubh-nair]

But I think we should turn on patch and put project off. Since patch will denote the test coverage of the pull request. And project is the one failing in those affected pull requests.

[kaustubh-nair]

Some files' coverage drops to 0%. Maybe there is some corruption with the reports? Maybe we could try using a different generator, if the problem still persists

[jywarren]

I think before "patch" was failing, now "project" is. Can you try opening a few PRs to test out more? It is weird! And just noting that not every PR will have tests, like HTML or CSS changes might not. Often this falls in the margin allowable like if we allow variances of up to 1% in coverage, you know?

On Sat, Jul 6, 2019 at 7:45 AM Kaustubh Nair [email protected] wrote:

Some files' coverage drops to 0%. Maybe there is some corruption with the reports? Maybe we could try using a different generator, if the problem still persists

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/publiclab/mapknitter/issues/810?email_source=notifications&email_token=AAAF6J57ZRLAWBCI3RC5RTDP6CASZA5CNFSM4H6NO6OKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZKYAHI#issuecomment-508919837, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAF6J75PAQ4TJQJWJ5Q7HTP6CASZANCNFSM4H6NO6OA .

[kaustubh-nair]

Reopening because we're still facing this in https://github.com/publiclab/mapknitter/pull/867 There's a possibility we're experiencing race conditions since we've got tests for models in both functional and unit tests, but I've got no idea how to verify that. https://github.com/colszowka/simplecov/issues/559

[kaustubh-nair]

@alaxalves @publiclab/mapknitter-reviewers Need some help with this issue. Haven't found anything to solve it after a lot of trying FYI this issue happens only on travis, and not locally. So there's a strong possibility it's related to splitting the tests and codecov is not properly merge those partial reports

[sashadev-sky]

@kaustubh-nair this is a complete shot in the dark but remember this custom rake task file mapknitter used to use to run its tests: https://github.com/publiclab/mapknitter/blob/main/lib/tasks/test_unit.rake

Perhaps you can use it to verify if the problem is a race condition - by running with Travis the different configurations it offers?

I remember it helped uncover a previous race condition issue in our tests months ago!

[kaustubh-nair]

Oooh might me a possibility! Let me have a look