Dmitry Frolov

icon company Staff SWE @ Lightning AI icon location New York, United States icon location @gridai @Lightning-AI @PyTorchLightning

Results 3 comments of Dmitry Frolov

The training hangs after reloading one of master/worker pods

I use pytorch 1.9.0.

The training hangs after reloading one of master/worker pods

I don't use it at the moment. I followed [mnist example](https://github.com/kubeflow/pytorch-operator/tree/master/examples/mnist) to adjust my training script.

GCP preemptible instances

The same question. How will PJ pods behave on instances which can stop/resume/reshedule workloads?