Deterministic mode sometimes show unexpected result.

[imsungwon]

Dear folks.

Thesedays I encounterd strange case where Octane benchmark shows incorrect result in case of non-deterministic mode.

Here is a simple example case.

ex ) minIteration = 4; device_A runs 4 times in 1sec. However, device_B runs 3 times in first iteration and couldn't satisfy minInteration and runs 7 times in additional iteration. Therefore, its final score becomes 5 in this case. device_A : 4/1 => 4 device_B : 10/2 => 5

I think this is unexpected situation since device_A runs faster for 1st iteration. device_A is better than device_B but deterministic mode show devices_B is better.

Can someone make it clear why Octane uses non-deterministic mode by default for some test cases?

Thank you in advance.