pax icon indicating copy to clipboard operation
pax copied to clipboard

Better progress indicator during multiprocessing

Open JelleAalbers opened this issue 8 years ago • 0 comments

When processing with several CPUS, the following information is shown in the status line:

  1. Overall status (input_done etc.)
  2. Number of events in queue between input and processing;
  3. Number of events in queue between processing and output;
  4. Memory usage split by input, workers, output.

3 is useless, since the output process always slurps the queue to put the events on a heap, which it regularly pops from to get the next event (this is needed for ensuring the order of events in the output). 4. is not really useful either, we haven't had memory leaks for some time, and you could get this info through htop anyway.

I would rather the status line shows us:

  • The last event number read/assigned by the input plugin, and the last event number written by the output plugin
  • Some indication of the speed (in events/sec)
  • If possible (ie when processing raw data from disk) an ETA.

Finally, the timing report is currently only printed in single-core mode, but for optimization it would be useful to combine this info from the multiple processes and dump it.

JelleAalbers avatar Jun 14 '16 09:06 JelleAalbers