no error reported by Hadoop in case of immediate failure

[klbostee]

As originally reported by Elias Pampalk:

The following scripts demonstrate a failure to fail when executed on a hadoop cluster (fails fine if executed locally):

import dumbo

def mapper(k, v):
    yield 1, 1

if __name__ == "__main__":
    dumbo.run(mapper, dumbo.sumsreducer, combiner=dumbo.sumsreducer)

The test uses dumbo.sumsreducer where dumbo.sumreduce should be used. A TypeError should be thrown by dumbo.sumsreducer (in the combiner). Instead the Hadoop reports show no error and zero output from the mapper.

[cap]

I encounter this non-failure whenever the script fails before calling dumbo.run. This happens most frequently when a top-level import statement fails because of unfulfilled dependencies.

[klbostee]

@cap: Think that's actually a slightly different problem. When a dumbo script fails very quickly (i.e. basically immediately, instead of after having run for a while) it often happens that Hadoop Streaming's stderr catching mechanism hasn't been set up properly yet to catch the error. In this case you indeed won't see the error, but unfortunately there's not much that can be done about this in Dumbo itself.

[klbostee]

On second thought this actually is the same problem. It does fail fine, but it happens so quick that the Hadoop Streaming logging doesn't catch it (presumably because it hasn't been initialized properly yet). Can't immediately think of a clean Dumbo-side fix for this problem, but I'll leave the ticket open for future reference...

[dangra]

I was bitten by this bug, a possible workaround is to delay the propagation of the exception by a sensible time so hadoop has a chance to setup streaming logging before terminating python interpreter. At least it will happen less often and will make dumbo more reliable at a low price.