InnerEye-DeepLearning
InnerEye-DeepLearning copied to clipboard
Jobs file when they get pre-empted after uploading files
Files get uploaded in run_ml.py register_model function via upload_folder
.
Then job gets pre-empted, starts again and tries to upload files again. At that point, it complains that files already exist on the run.
To fix:
- Check if the error message indicates that all or only some files already exist
- Rather than using upload_folder, upload individual files that are missing from the run. Should achieve the same performance because upload_folder uses upload_files under the hood.