Tests refactoring

[Borda]

🚀 Feature

thinking about https://github.com/PyTorchLightning/metrics/pull/676#issuecomment-991161059 raised by @lucadiliello we can make an assumption that there shall not be too many differences among the Python versions and OS, right? in such a case we can have three lines of testing:

• doctest on the package [we have now] which is testing examples in docs and imports
• some general and most common tests [need to be specified] which would be reasonably fast and run and we run it on all configurations as it now
• running full tests [as it is now] on Linux all python versions

Motivation

Boost the CI response but still, keep high test coverage.

Pitch

Alternatives

We can also think to run some limited testing on PRs and then set command which would before merging run some extensive testing. This extensive testing would also run on commit merge master

Additional context

Thoughts?

cc: @PyTorchLightning/core-metrics

• #675
• #733

[SkafteNicki]

In my experience, most times the difference lies in the Pytorch version that is being tested where something may work for version (x+1) but not for version x. So I would still do full Pytorch testing on each PR.

[Borda]

lest also explore auto-canceling via group: https://github.com/marketplace/actions/auto-cancel-redundant-workflow

[twsl]

concurrency is preferred because the auto-cancel repo is no longer maintained

[Borda]

we can benefit from principle that domains are independent, so we can for each PR determine if it is related only to domain level (adding new NLP metric) or it affects all metrics (for example changes in the base Metrics class)

[Borda]

Another point for discussion about static testing references #949