metaflow icon indicating copy to clipboard operation
metaflow copied to clipboard
verified publisher icon Netflix

Reduce the memory overhead of artifact serialization

Open tuulos opened this issue 5 years ago • 1 comments

Artifact serialization at the end of a task creates a copy of each artifact in memory for a short while. This is problematic especially locally when you have N tasks serializing artifacts in parallel.

We should look into reducing the overhead. Context https://gitter.im/metaflow_org/community?at=5f7bdb663d3d0d749a307ea3

tuulos avatar Oct 06 '20 02:10 tuulos

Is there any context to this issue available any more? The original gitter link seems to not be publicly accessible, it is possible to reproduce it here on on the Slack Channel?

stev-0 avatar Mar 06 '24 11:03 stev-0