whateverable
whateverable copied to clipboard
Raku
CI test?
Yes! But it is probably not as easy as it sounds. The situation is that I just managed to get this thing runnable locally (by pulling the required builds from the server), but the test suite is not designed for the extra time it takes to download the builds (note that these tests are acting as a shotgun for almost all imaginable issues, so it's system tests only where each test is running some query on an actual bot with real data, through a real local IRC server).
So to make that work I think something like this should be used: https://github.com/perl6/whateverable/blob/50f4a0204ebb4dd4fcb1d6a6083eba1819ba1627/lib/Whateverable.pm6#L359-L364 Somewhere around here: https://github.com/perl6/whateverable/blob/50f4a0204ebb4dd4fcb1d6a6083eba1819ba1627/lib/Whateverable.pm6#L284-L292 This way, the bot can signal the test suite that it needs some extra time for the test.
If anybody is willing to work on this I'd be grateful, but the code is not ready yet for proper CI.
Oh, also:
It is very common for test suites or build scripts to hang. Travis CI has specific time limits for each job, and will stop the build and add an error message to the build log in the following situations:
- A job on travis-ci.org takes longer than 50 minutes
When running these tests sequentially it takes more than 30 minutes I think. There are lots of tests, and many of them need some quality time with the CPU. There were attempts to parallelize it, and that should be revisited I think.
We could do several different things.
- Create our own CI environment using, for instance, bazel.build. We would have to host it ourselves.
- Separate the tests in xt and t, and run only t. t would include most of the tests that would be sensible to do given pull requests.
- Try another environment that gives us 50 minutes.
- Reduce testing time by using your binary :-) That might cut it down by 15 minutes.
Parallelizing would be a good idea, but that would take us to create a Test::Concurrent module and a good amount of yak shaving... :-)
Also creating a Docker container for testing would save additional time by having all needed libraries already included. I could try that anyway.
- Reduce testing time by using your binary :-) That might cut it down by 15 minutes.
It's not so much about the setup time, but the sheer amount of tests and what they do. As of today, it takes ≈12 seconds to bisect something in the default range (2015.12⌁HEAD), but it also depends on some other factors. And bisectable.t has many of queries like this. :)
Parallelizing would be a good idea, but that would take us to create a Test::Concurrent module and a good amount of yak shaving... :-)
It's actually quite easy given that you can ask prove to run test files in parallel. In fact, sake test will already do that by default:
https://github.com/perl6/whateverable/blob/50f4a0204ebb4dd4fcb1d6a6083eba1819ba1627/Sakefile#L41
Also creating a Docker container for testing would save additional time
Would be nice if we can do without docker containers, but it's possible that we wouldn't be able to get it working any other way. Just look at all these deps 🤢: https://github.com/perl6/whateverable/blob/50f4a0204ebb4dd4fcb1d6a6083eba1819ba1627/META6.json#L12-L33
@JJ I think we should make separate targets for every single bot. Latest rakudo should be pulled similarly to how it's done in the doc repo. Modules can be installed with --/test so that we save some extra time.
OK, we'll try that. I'm kind of busy right now (not to mention elsewhere) but I'll get to that as soon as I can.