gdshelpers icon indicating copy to clipboard operation
gdshelpers copied to clipboard

Allow cell reuse when creating cells in parallel

Open fbeutel opened this issue 2 years ago • 0 comments

Sometime, the cell creation is expensive and benefits from parallelization. At the same time, some sub-cells can be reused between cells.

Currently, this fails because when ProcessPoolExecutor (or similar parallelization structures) are used, sub-cells are pickled and deserialized again. When merging the parallel cells together in the end, comparison for those sub-cells fails because they have no common identity anymore.

This adds a UUID when a cell is created, such that cells can be identified by their UUID.

Possible downside is that if a user modifies a sub-cell after it has been pickled, the changes might get lost, but this can easily be avoided by finishing the sub-cell before using it.

fbeutel avatar Jul 29 '21 09:07 fbeutel