tbvaccine icon indicating copy to clipboard operation
tbvaccine copied to clipboard

TBVaccine produces insane reporting when Dask or Dask Distributed task crashes

Open frol opened this issue 7 years ago • 0 comments

Here is the reproduction:

import tbvaccine
tbvaccine.add_hook(isolate=False)

import dask

def x(arg1=123):
    assert arg1 == 122, "qwe"

if __name__ == '__main__':
    dask.delayed(x)().compute()

The error:

  File "/tmp/env/lib/python3.5/site-packages/dask/async.py", line 272, in _execute_task
    return func(*args2)
  File "qq5.py", line 7, in x
    assert arg1 == 122, "qwe"
|     worker_id                = None
AssertionError: qwe

I don't understand at all where worker_id came from.

Running Dask Distributed, I see the output, which reports things which don't exist in that scope at all...

import tbvaccine
tbvaccine.add_hook(isolate=False)

import dask
import dask.distributed


def x(arg1=123):
    assert arg1 == 122, "qwe"

if __name__ == '__main__':
    client = dask.distributed.Client()
    dask.delayed(x)().compute(get=client.get)
  File "/tmp/env/lib/python3.5/site-packages/dask/base.py", line 203, in compute
    results = get(dsk, keys, **kwargs)
|     x = b'\x80\x04\x95\xa1\x00\x00\x00\x00\x00\x00\x00\x8c\x16tblib.pickling_support\x94\x8c\x12unpickle_traceback\x94\x93\x94\x8c\x05tblib\x94\x8c\x05Frame\x94\x93\x94)\x81\x94}\x94(\x8c\x06f_code\x94h\x03\x8c\x04Code\x94\x93\x94)\x81\x94}\x94(\x8c\x07co_name\x94\x8c\x01x\x94\x8c\x0bco_filename\x94\x8c\x06qq4.py\x94ub\x8c\tf_globals\x94}\x94ubK\tN\x87\x94R\x94.'
  File "/tmp/env/lib/python3.5/site-packages/distributed/client.py", line 1590, in get
    resources=resources)
|     ret       = <tblib.Traceback object at 0x7f95aeadfcf8>
|     tb_frame  = <tblib.Frame object at 0x7f95aeadf908>
|     tb_lineno = 9
|     tb_next   = None
  File "/tmp/env/lib/python3.5/site-packages/distributed/utils.py", line 223, in sync
    six.reraise(*error[0])
|     code   = <code object x at 0x7f95aeb33270, file "qq4.py", line 9>
|     f_code = <tblib.Code object at 0x7f95aeadf9b0>
|     self   = <tblib.Traceback object at 0x7f95aeadfcf8>
|     tb     = <traceback object at 0x7f95aeaeb148>
  File "/tmp/env/lib/python3.5/site-packages/six.py", line 686, in reraise
    raise value
AssertionError: qwe

The direct call to x() produces a sane output, so it should be some magic involved with Dask. Better-Exceptions module just hangs in this situation, so TBVaccine is better in this respect, but it is still completely misleading and unhelpful.

frol avatar Jun 01 '17 11:06 frol