[BUG] equality_check fails when a kernel has reached its cutoff

[pdemarti]

Describe the bug

A nice feature of perfplot is to be able to compare results that take dramatically different times. Imagine one is O(n) and another O(n!). In that case, using the parameter max_time cuts off a kernel once its measurements get larger than max_time over the course of n_range.

However, the timings produced for that kernel after cutoff are NaN. This is fine (even desired) for the benchmark results and for plotting, but alas, equality_check is called even though there is no result for that kernel.

To Reproduce

from math import factorial
import time

def f(n):
    time.sleep(0.001 * factorial(n))
    return 1

def g(n):
    time.sleep(0.001 * n)
    return 1

perfplot.plot(
    setup=lambda n: n,
    n_range=range(6),
    kernels=[f, g],
    max_time=0.005,
)

# raises: TypeError: ufunc 'isfinite' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
# and: PerfplotError: Error in equality_check. Try setting equality_check=None.

Note: to make it work (with equality check), we can invert the order of the kernels (s.t. kernels[0] doesn't reach cutoff), or alternatively define a custom equality_check that handles the presence of None:

    ...
    equality_check=lambda a, b: a is None or b is None or a == b,

Diagnose I may ask you to cut and paste the output of the following command.

conda list perfplot
# packages in environment at /mnt/miniconda3/envs/py39:
#
# Name                    Version                   Build  Channel
perfplot                  0.10.2             pyh6c4a22f_0    conda-forge

Did I help?

If I was able to resolve your problem, consider sponsoring my work on perfplot, or buy me a coffee to say thanks.