Javier

Restarting training from checkpoint on large model goes out of memory

1

Training a model with this params: --model=HeadAndNeck --train=True --cluster=training-nd40-v2 --run_recovery_id=jaalvare_bigcluster:jaalvare_bigcluster_1607291058_70ac48b7 --start_epoch=120 Results on: 2021-01-05T13:13:20Z ERROR Model training/testing failed. Exception: CUDA out of memory. Tried to allocate 12.74 GiB (GPU 0;...

The AzureML should fail and send an email if model comparison fails. This can help with regression testing. [AB#5093](https://innereye.visualstudio.com/60ce1777-00d6-4015-82bc-488a0c00202f/_workitems/edit/5093)

https://pypi.org/project/vulture/#:~:text=Vulture%20finds%20unused%20code%20in%20Python%20programs.%20This,Vulture%20are%20likely%20to%20miss%20some%20dead%20code. [AB#3953](https://innereye.visualstudio.com/60ce1777-00d6-4015-82bc-488a0c00202f/_workitems/edit/3953)

[AB#3931](https://innereye.visualstudio.com/60ce1777-00d6-4015-82bc-488a0c00202f/_workitems/edit/3931)

- GPU augmentations at training time should be configurable - Document the benefits of this feature [AB#3925](https://innereye.visualstudio.com/60ce1777-00d6-4015-82bc-488a0c00202f/_workitems/edit/3925)

It is non transparent if the datasets are exactly the same version or we are just comparing patient ids. More debugging info in general would be good. [AB#3910](https://innereye.visualstudio.com/60ce1777-00d6-4015-82bc-488a0c00202f/_workitems/edit/3910)

Azure AssetStore in Digital slide archive

6

Would you like to receive a pull request to support Azure blob in your AssetStore? Thanks, Javier

Move to python 3.10

1

Python 3.7 only has 1 year until end of life: https://endoflife.date/python

Pytorch lightning

2

Do you know if ort works with Pytorch lightning? I am trying it but I am getting: raise new_exception(raised_exception) from raised_exception onnxruntime.training.ortmodule._fallback.ORTModuleTorchModelException: ORTModule does not support adding modules to it....