earth2mip icon indicating copy to clipboard operation
earth2mip copied to clipboard
verified publisher icon NVIDIA

🐛[BUG]: Race condition in parallel workflows

Open nbren12 opened this issue 2 years ago • 1 comments

Version

main

On which installation method(s) does this occur?

No response

Describe the issue

The model data is downloading by all ranks of a parallel job in parallel. This will result in corrupt files if the files haven't been download into the model registry yet.

Environment details

No response

nbren12 avatar Oct 31 '23 20:10 nbren12

@yairchn Just ran into this problem. worth fixing.

nbren12 avatar Dec 19 '23 22:12 nbren12