kedro-rich icon indicating copy to clipboard operation
kedro-rich copied to clipboard

`ParallelRunner` doesn't work yet

Open datajoely opened this issue 3 years ago • 1 comments

Description

Due to the way the ParallelRunner works in Kedro via multi-processing the existing hook implementation breaks when interacting with whatever UI thread rich needs to work.

Context

Currently if the user specifies parallel runner the progress bar logic is disabled.

Potential solutions to evaluate:

  1. Work out a way of making the existing implementation using Locks (preferred)
  2. Subclass the ParallelRunner implementation and do it there (not ideal)

datajoely avatar Mar 07 '22 09:03 datajoely

I can't get the progress bars to work with ParallelRunner, but the nicer logging experience still works. Ultimately without developing my own runner implementation, I think the the interference of the threading rich is using behind the scenes and the multiprocessing Kedro is doing just won't work nicely together.

datajoely avatar Mar 18 '22 18:03 datajoely