deepforge
deepforge copied to clipboard
Operations should be batched when not parallelizable
To reduce data transmission across the network, operations in executing pipelines should be batched
It is actually not clear that this is how it should behave. This will result in not saving the intermediate data between operations (making it so you can't necessarily restart just a single job).
However, if we are not in debug mode, then this might make sense as the actual operations won't be edited and restarted (as they are snapshots and cannot be edited).