CoconutSweet999

fatal error: nccl.h: No such file or directory

1

**Environment:** 1. Framework: (TensorFlow, Keras, PyTorch, MXNet) 2. Framework version:PyTorch 3. Horovod version:0.28.0 4. MPI version:4.0.7 5. CUDA version:11.4 6. NCCL version:2.11.4 7. Python version:3.7 8. Spark / PySpark version:...

**Environment:** 1. Framework: (TensorFlow, Keras, PyTorch, MXNet) 2. Framework version:PyTorch 3. Horovod version: 4. MPI version:4.1.5 5. CUDA version:12 6. NCCL version: 7. Python version: 8. Spark / PySpark version:...

About the loss: reg loss and cls loss

3

Hi!Thanks for sharing your code！ When I tried to retrain the model, I found that the reg loss gradually dropped to negative numbers, while the cls loss first went down...

How to load checkpoint and continue training?

1

I wonder how to load checkpoint and continue training. Because of the OOM (maybe at 30 epochs), I have to load checkpoint and continue training after OOM. I try to...