legion icon indicating copy to clipboard operation
legion copied to clipboard

slow but correct 1000-job test case causes failure

Open hraban opened this issue 3 years ago • 1 comments

Hi,

The test in https://github.com/fukamachi/legion/blob/master/t/legion.lisp#L135 hard-codes a 1-1.3× duration as "test success", and anything slower as "test failure".

On my machine, a macbook pro 2015, it can easily reach 10x:


 process 1000 jobs
    × Expected it to end in 5.0 - 6.5 seconds, but took 44.761331d0
      NIL is expected to be T

Should this actually be considered a failing test, though? I don't mind that it's slow, as long as it works. Where does the 1.3x come from? Shouldn't this be a benchmark rather than a unit test?

hraban avatar Dec 04 '22 20:12 hraban

I guess my worry is that this makes the correctness of the test dependent on the machine running the test, which doesn't seem ideal.

hraban avatar Dec 04 '22 20:12 hraban