fix: combined coverage report

[rudaporto]

Currently, we have some issues regarding the combined coverage:

• we have two sections in the tox.ini to check the coverage: one using only Python 3.8 and one extra combine all the coverage generated from each python version
• the combined-coverage section does not work because of the order of the coverage section that runs before
• the coverage section is used in the CI together with coveralls but it only checks with one Python version meaning it can not enforce it to be 100% because i.e: https://coveralls.io/jobs/107753481

This PR is a POC to try to find a better way to implement both, the local coverage check and the CI one.

** I',m aware that these files are autogenerated by the meta package. If we agree that this makes sense I would like to propose the creation of new templates in the meta package to handle this use case. **

Here follow all the changes I implemented:

Tox:

• coverage env should always be the last
• default coverage should be enough (no need for the extra combined-coverage config)
• fix the coverage env to do the combine and fail under 100%

CI:

• build: remove the coverage from the matrix since it will become a separate job
• build: add a new upload artifact step to run after tests
• coverage: a new job that depends on the build job
• coverage: add a step to download all artifacts uploaded from the build steps
• coverage: add a step to combine the coverage artifacts and upload them to coveralls
• coverage: add a step to build the report and check if coverage matches 100%
• coverage (optional): add a step to upload the HTML report for when the job failed

Next step:

• verify if this is what we want and improve and polish if needed
• add support in the meta for combined coverage as a new template and options (see issue already open before https://github.com/zopefoundation/meta/issues/33)
• update the .meta.toml to use the new template/options and generate the files creating the final PR

[rudaporto]

When using the matrix execution we can not collect the combined coverage from each python version.

[davisagli]

https://hynek.me/articles/ditch-codecov-python/ has a suggestion of how to do this

[dataflake]

FYI, the file .github/workflows/tests.yml is auto-generated and should not be edited by hand. It uses the meta/config package, see https://github.com/zopefoundation/meta/tree/master/config, which is configured using the .meta.toml file.

[rudaporto]

FYI, the file .github/workflows/tests.yml is auto-generated and should not be edited by hand. It uses the meta/config package, see https://github.com/zopefoundation/meta/tree/master/config, which is configured using the .meta.toml file.

Thank you @dataflake for the reminder. Yes, I know that this should not be changed directly. I've updated the description with a full explanation of what I tried to achieve.

I will convert this PR to a draft to avoid confusion.

[icemac]

I think this approach presented here is promising. It requires some changes in meta/config so it can produce the required tests.yaml. Currently we are running coverage just for one Python version. In most projects this is enough because (after dropping Python 2 support) there is nearly no Python version specific code. So for the other projects I think it will be enough just to run coverage on a single Python version, but here we have different goals.

I currently do not have the time and energy to push meta/config forward to be able to support the kind of changes required here, sorry.