azure-pipelines.yml: test important packages

[alaviss]

[dom96]

Do we need another CI for this? Can't we do it in travis?

[dom96]

To be honest, I wonder whether this should even be done every commit. It seems like it would be very flaky. We should have it run continuously every day or so instead to give us general signal, but every commit sounds like a pain.

[alaviss]

Do we need another CI for this? Can't we do it in travis?

It usually takes over 10 mins for an output from testament (I guess it was doing all the cloning in the background), which just cause travis to terminate due to timeout.

To be honest, I wonder whether this should even be done every commit. It seems like it would be very flaky. We should have it run continuously every day or so instead to give us general signal, but every commit sounds like a pain.

It appears to me that nimble has been failing the "important packages test" for a while: https://dev.azure.com/alaviss/Nim/_build/results?buildId=280&view=logs&j=12f1170f-54f2-53f3-20dd-22fc7dff55f9&t=356bb04c-cb4a-5f04-82ca-d3b102917eba

I can reproduce the errors, and they aren't there with stable nimble (which is used by Nim devel for testing). So yes, I think we should do this kind of testing more often.

[dom96]

If we can get a list of packages that Nimble master fails on but where the last stable Nimble succeeds then that would be very high signal. But setting up and navigating all these different kinds of CIs is quite frustrating.

[timotheecour]

It seems like it would be very flaky.

if it becomes too flaky we can simply add:

bash: testament cat nimble-packages

=>

bash: testament cat nimble-packages || echo "FAILED (allowing failure)"

but this is strictly better than not testing at all

[dom96]

It can give good signal, but I'd rather it not block PRs/commits which is why I think it should be run separately. Also, I would prefer if it didn't depend on testament.

[timotheecour]

it wouldn't block anything because of the || echo "FAILED (allowing failure)" which causes it to never fail; if by block you mean "takes too long" (in practice should take as long as it does in nim repo, ie ~32mn), you can always decide to squash&merge a commit without waiting for it to complete.

[dom96]

That could work but only if we can avoid noise in GitHub's UI.

[timotheecour]

as long as a task succeeds (eg via || echo "FAILED (allowing failure)"), IIUC github UI would just show successful check, ie no noise, but logs could still be inspectable for failures

[dom96]

We now have Github Actions, can we do it there?

[dom96]

To be honest, I'm still preferring a separate repo for this so that we can avoid delaying the CI status.