clustermq icon indicating copy to clipboard operation
clustermq copied to clipboard

Local processing reports errors only after all calls complete

Open mschubert opened this issue 10 months ago • 1 comments

Consider the following function

fx = function(x) {
    if (x %% 2 == 0)
        warning(paste("warning on", x))
    if (x %% 3 == 0)
        stop(paste("error on", x))
    x
}

clustermq::Q(fx, 1:10, n_jobs=1)

This fails correctly after 3 calls:

Starting 1 cores ...
Running 10 calculations (4 objs/4.3 Kb common; 1 calls/chunk) ...
Error: 1/3 jobs failed (1 warnings). Stopping.
(Error #3) error on 3
(#2) warning on 2

However, if running with n_jobs=0, all calls are evaluated before the error is evaluated:

Running sequentially ('LOCAL') ...
Error: 3/10 jobs failed (5 warnings). Stopping.
(Error #3) error on 3
(Error #6) error on 6
(Error #9) error on 9
(#2) warning on 2
(#4) warning on 4
(#6) warning on 6
(#8) warning on 8
(#10) warning on 10

This should probably exit early if fail_on_error=TRUE. Ideally, we would also support a custom error handling function supplied by the user (related: #164)

mschubert avatar Aug 23 '23 13:08 mschubert