long tracebacks

[jrmlhermitte]

I'm wondering if we can avoid long tracebacks:

In [1]: from streamz import Stream

In [2]: s = Stream()

In [3]: snext = s
   ...: for i in range(10):                                                                     
   ...:     snext = snext.map(lambda x : x)
   ...:     

In [4]: def foo(x):
   ...:     raise ValueError("Foo")                                                                               
   ...: 

In [5]: snext.sink(foo)
Out[5]: <sink: foo>

In [6]: s.emit(1)
.... Long error message
<ipython-input-4-9636b5287c3d> in foo(x)
      1 def foo(x):
----> 2     raise ValueError("Foo")

ValueError: Foo

A way around this could be that the stream returns a graph which is executed one by one by an external function, like:

graph = s.to_graph()
run(graph, input)

where the run function would walk through each node in the graph in a top-down fashion and resolve the results.

Any other thoughts? Just an idea I figured I'd document somewhere. thanks!

[CJ-Wright]

For me this issue is a bit broader, debugging complex pipelines can be rough. It might be nice to have ways to make it less rough.

Another approach could be to run function based nodes (sink, map, filter, etc.) with a try, except around the function call. That way we can raise a nice exception which locates which node generated the exception, what data it was generated with and such.

Another approach could be to write a helper function which subscribes a specific debug print sink to every node in the graph. The print would print the node name, input/output data, etc. This way offending outputs could be found and the node modified.