dora icon indicating copy to clipboard operation
dora copied to clipboard

No Input node does not return `STOP` event

Open haixuanTao opened this issue 1 year ago • 2 comments

Describe the bug Python Node with no input always return None on .next() although we would want it to return STOP event so that we can gracefully stop it.

import pyarrow as pa
from dora import Node

node = Node()

node.send_output("data", pa.array([1, 2, 3, 4, 5]))

event = node.next()

assert event is not None, "we should expect a STOP event"
assert event["type"] == "STOP", "we should expect a STOP event"
nodes:
  - id: no_input
    path: no_input.py
    outputs:
      - data

  - id: terminal-print
    build: cargo build -p terminal-print
    path: dynamic
    inputs:
      data: no_input/data

haixuanTao avatar Aug 15 '24 05:08 haixuanTao

That is expected with our current design. The event channel returns None as soon as all inputs of the node have been closed. This allows nodes to stop when they are no longer needed. The Stop event signals that the dataflow was stopped early as a consequence of a stop command. So it signals that the event channel might close sooner than usual.

We could of course special-case nodes with no inputs and keep their event channels open as long as the dataflow runs. However, such behavior differences would be inconsistent and surprising, and also difficult to document clearly. Another disadvantage is that next() is a blocking, so that a call without a timeout would block until the user cancels the dataflow.

How about we provide an additional dataflow_stopped function instead? This function could be non-blocking and return true as soon as the manual stop signal was received by the node. So it would be suitable for checking it a loop.

phil-opp avatar Aug 19 '24 10:08 phil-opp

That would be great! The name could be just finished() slightly shorter maybe?

haixuanTao avatar Aug 21 '24 14:08 haixuanTao