redis-rb

redis-rb copied to clipboard

Description Adds TruffleRuby to GitHub Actions ruby versions to test for compatibility.

It's over 10 times slower than the regular rubies, so it's a hard sell. If you find a way to make it faster I would consider it, but in this state I'm really not convinced. I'm also not a fan of running truffle-head, as GitHub action doesn't support allowed failures AFAIK.

15min vs 2min30s (for DRIVER=ruby) that seems too much indeed. I think we should look into why it's much slower there.

We could use a release (there is one in a month, feature freeze is this week, https://www.graalvm.org/release-notes/version-roadmap/), but if we do performance improvements then they will likely not be there. step-level continue-on-error is a way to achieve some kind of allowed failure. job-level continue-on-error is closer, but the reporting is confusing (it marks the job as failed even though it's marked as allowed to fail, and the whole workflow as passing, but the PR UI does not show workflow status, only jobs, https://github.com/actions/runner/issues/2347).

I just tried this again with TruffleRuby 22.2 https://github.com/redis/redis-rb/runs/7885051141?check_suite_focus=true

Took 9m where JRuby takes 4m30 and MRI 4m (I'm talking just the test run, not the setup).

So it's much better than before, but I'm still really on the fence as doubling CI time is a hard sell :/

I'll take a look if there is an easy win. One easy thing to try is using TRUFFLERUBYOPT=--engine.Mode=latency.

I've got 7min23s on https://github.com/eregon/redis-rb/runs/7887133053?check_suite_focus=true with TRUFFLERUBYOPT=--engine.Mode=latency. (I also did a 2nd run with your commit, https://github.com/eregon/redis-rb/runs/7887116546?check_suite_focus=true, 8m56s)

IMHO that's already pretty fast for CI, so I think it'd make sense to merge that.

TruffleRuby does more profiling than CRuby/JRuby in the interpreter, and we can't expect to run in compiled code the whole time during tests (since there is lots of new code being loaded while running tests, few CPUs to JIT, and lots of edges cases discovered through the test suite which can lead to recompilations) so it seems reasonably close to me.

What we could do further is to profile it, but we can't change 22.2.0 of course, so that'd need to wait the next release (or use truffleruby-head), which feels like waiting unnecessarily.

IMHO that's already pretty fast for CI,

Well, that's a subjective thing :p, 4-ish minutes to 7m23 is almost twice slower, but that's already much better so I can take that.

I can also split the regular and distributed suites to parallelize that, so it's fine.

Thanks!