Neuraxle icon indicating copy to clipboard operation
Neuraxle copied to clipboard

Feature: QueuedPipelines should be able to handle exceptions properly

Open alexbrillant opened this issue 5 years ago • 1 comments

Is your feature request related to a problem? Please describe. Workers should be able to catch exceptions properly and broadcast a termination signal to all of the current workers.

First thoughts about this ?

  1. Add a broadcast method to ObservableQueueMixin that sends a termination signal to all of the observers. https://github.com/Neuraxio/Neuraxle/blob/7cf9460eb2e971b32b4b84b8441d8515c9247d83/neuraxle/distributed/streaming.py#L90
  2. Broadcast exception the except block of the worker_function here: https://github.com/Neuraxio/Neuraxle/blob/7cf9460eb2e971b32b4b84b8441d8515c9247d83/neuraxle/distributed/streaming.py#L251
  3. Handle Continue exception in the QueueJoiner : https://github.com/Neuraxio/Neuraxle/blob/7cf9460eb2e971b32b4b84b8441d8515c9247d83/neuraxle/distributed/streaming.py#L713
  4. Handle the termination signal in the main thread to kill all processes and clean the workers....

Think about it that might not be the right solution...

alexbrillant avatar Oct 08 '20 22:10 alexbrillant

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs in the next 180 days. Thank you for your contributions.

stale[bot] avatar Apr 11 '22 22:04 stale[bot]