AttributeError: 'NoneType' object has no attribute 'enabled'

[mugdha-t]

Hello, I was doing transfer learning on yolov5l pruned+quant model using ddp multi gpu setup. It almost completed 49/51 epochs and stopped with the following error. I was using weights downloaded from the zoo as I got a server error using the zoo stub.

Steps to reproduce:

conda create -n venv python==3.8
git clone https://github.com/neuralmagic/yolov5.git

python3 -m torch.distributed.run --nproc_per_node 4 train.py --data ~/data/data_fixed.yaml --cfg ~/sparseml/integrations/ultralytics-yolov5/models_v5.0/yolov5l.yaml --weights ~/zoo-yolov5l-p-q.pt --hyp ~/sparseml/src/sparseml/yolov5/data/hyps/hyp.finetune.yaml --recipe ~/sparseml/integrations/ultralytics-yolov5/recipes/yolov5.transfer_learn_pruned_quantized.md --batch-size 16 --device 0,1,2,3

And the stacktrace:

File "train.py", line 761, in <module>                                                                                                                   
Traceback (most recent call last):                                                                                                                         
  File "train.py", line 761, in <module>                                                                                                                   
Traceback (most recent call last):                                                                                                                         
  File "train.py", line 761, in <module>                                                                                                                   
    main(opt)                                                                                                                                              
  File "train.py", line 657, in main                                                                                                                       
    main(opt)                                                                                                                                              
  File "train.py", line 657, in main                                                                                                                       
    main(opt)                                                                                                                                              
  File "train.py", line 657, in main                                                                                                                       
        train(opt.hyp, opt, device, callbacks)train(opt.hyp, opt, device, callbacks)                                                                       
                                                                                                                                                           
  File "train.py", line 363, in train                                                                                                                      
  File "train.py", line 363, in train                                                                                                                      
    train(opt.hyp, opt, device, callbacks)                                                                                                                 
  File "train.py", line 363, in train                                                                                                                      
    ema.enabled = False                                                                                                                                    
AttributeError: 'NoneType' object has no attribute 'enabled'                                                                                               
    ema.enabled = False                                                                                                                                    
AttributeError: 'NoneType' object has no attribute 'enabled'                                                                                               
    ema.enabled = False                                                                                                                                    
AttributeError: 'NoneType' object has no attribute 'enabled'