Retag "expensive" tests

[sschuberth]

    > Sounds good. So with #5865 in place, can this be merged now?

Not yet. #5865 only enabled the build scans for tests to get the durations of tests. We'd still need to agree on from when on a test counts as "expensive", retag all tests accordingly, exclude expensive tags from PR runs, and set up a new job that runs expensive tests on a nighly basis. I don't have the time to work on all of this.

To get a feeling for recent test durations, see

• (unit) tests
• non-analyzer functional tests
• and I just realized I missed to enable build scans for analyzer functional tests (will do that now)
  • edit: see https://github.com/oss-review-toolkit/ort/pull/5869 which produces analyzer functional tests

But in order to get your fixed merged, can we either disable the test, or split it out to a separate PR? And while doing that, we could also consider to split the addition of the rather big test project from the addition of the test itself.

Originally posted by @sschuberth in https://github.com/oss-review-toolkit/ort/issues/5857#issuecomment-1259040123

[sschuberth]

@oss-review-toolkit/kotlin-devs from what duration on should we regard a single test case as "expensive"? 30s? 1m? For our current test durations please see the links above.

[mnonnenmacher]

I'm not sure, the tests at he top of the list are all tests I would prefer to run on each change, I often rely on the CI tests because it's sometimes hard to know which tests might be affected by a change. Maybe we should instead look into the specific test cases and check if they can be made faster, for example by reducing the size of the test projects.

[sschuberth]

for example by reducing the size of the test projects.

Apparently that wasn't possible for a recently added test. So how about instead removing the ExpensiveTag (as we're not using / maintaining it away) and instead introduce a NightlyTag for tests that we want to only run once a night / day (for whatever reason)?

[mnonnenmacher]

I like NightlyTag because it's more specific than ExpensiveTag.

[sschuberth]

I like NightlyTag because it's more specific than ExpensiveTag.

Actually I think the opposite: ExpensiveTag better expresses why you'd want to run a test nighty / daily / not per PR.

[sschuberth]

Closed as part of backlog grooming. Feel free to comment if you would like to contribute to this.