strax icon indicating copy to clipboard operation
strax copied to clipboard

Adjust timeout automatically to amount of returned time / processing speed

Open JelleAalbers opened this issue 4 years ago • 0 comments

Each plugin's mailbox times out processing when it has received no new data for too long, suspecting strax might be in deadlock.

The amount of time per chunk can very greatly across datatypes, especially when they are loaded from disk. Some small high-level ones might have only one chunk per run, while big low-level ones have one every few seconds.

Thus, if processing uses a one-chunk datatype, the timeout must currently be as long as the entire expected duration of the processing, or it causes unnecessary crashes. These will show up as e.g.

strax.mailbox.MailboxReadTimeout: <Mailbox: peaklet_classification_mailbox> could not progress beyond 1, no driving subscriber requested it

For now, I'm going to set the timeout to some silly high number, e.g. 24 hours. Effectively this means we don't have protection from hangs anymore.

It would be good to adjust the timeout per datatype to the amount of time per chunk and the processing speed somehow. Or replace the mailbox system altogether, see #81.

... or, we could add a 'global' timeout, which checks how long we see nothing new in any mailbox at all.

JelleAalbers avatar Apr 13 '20 12:04 JelleAalbers