support per-PR configurable statuses?

[cgwalters]

trafficstars

Homu was started with the idea that the master branch should pass "all" of the tests. However, for some projects, one really wants different definitions of "all".

The "rollup" model admits that some PRs (e.g. to docs) aren't going to break anything, so should just be aggregated with others and you only have one potentially expensive test run.

Many people configure their build system (e.g. Jenkins) to listen for optional text like "bot, run extended tests" - basically one status, but the test sometimes does more expensive things.

It might be interesting however to have Homu know about multiple statuses, and optionally tell Homu to watch for them.

Something like:

@homu test=extended
@homu r+ a12bc3d

The advantage of this is that one can clearly see in the PR which test suites were run.

[larsbergstrom]

When I was at Microsoft, our circa-2005 autolander bot infra would use code coverage info for changes to determine the minimum set of tests required to land a change. People could additionally opt into more tests via similar test requests (in this case, it was a huge web form rather than issue comments, but the idea's the same). We've assumed we will eventually need such a system for Servo, so I'd certainly be supportive of landing similar support pieces over time.

That said, it's a bit at philosophical odds with http://graydon.livejournal.com/186550.html, so I'm sure any such changes would be contentious.

[Manishearth]

Well, the idea is not to cut down on tests, it's to add certain tests that don't need to be run often (nightly and on certain PRs, perhaps).

Sadly you can't have travis report multiple statuses so you need to find a new CI thing for each extra test for this to work.

However, we could have @homu message="FOOBAR", and FOOBAR is added to the merge commit message, which can be detected by your CI script.