InnerEye-DeepLearning icon indicating copy to clipboard operation
InnerEye-DeepLearning copied to clipboard

Jobs file when they get pre-empted after uploading files

Open ant0nsc opened this issue 3 years ago • 0 comments

Files get uploaded in run_ml.py register_model function via upload_folder. Then job gets pre-empted, starts again and tries to upload files again. At that point, it complains that files already exist on the run.

To fix:

  • Check if the error message indicates that all or only some files already exist
  • Rather than using upload_folder, upload individual files that are missing from the run. Should achieve the same performance because upload_folder uses upload_files under the hood.

AB#4525

ant0nsc avatar Sep 27 '21 08:09 ant0nsc