save_best=True while encountering empty result

[Chan-Sun]

Describe the bug Hi! I was deploying mmfewshot detection model on my own dataset. And "save_best=True" was added in evaluation dict. Error happens for ckpt saving when empty result come out during validation stage

Error traceback If applicable, paste the error trackback here.

here are some logs:

/home/user/anaconda3/envs/sc_mmlab/lib/python3.7/site-packages/mmcv/runner/hooks/evaluation.py:375: UserWarning: Since `eval_res` is an empty dict,
ior to save the best checkpoint will be skipped in this evaluation.                                                                                          
  'Since `eval_res` is an empty dict, the behavior to save '                                                                                                 
Traceback (most recent call last):                                                                                                                             
File "./Main/mmfewshot/tools/detection/train.py", line 215, in <module>                                                                                    
    main()                                                                                                                                                     
    File "./Main/mmfewshot/tools/detection/train.py", line 211, in main                                                                                        
    meta=meta)                                                                                                                                                 
    File "/home/user/sun_chen/Projects/FsDefect/Main/mmfewshot/mmfewshot/detection/apis/train.py", line 197, in train_detector                                 
    runner.run(data_loaders, cfg.workflow)                                                                                                                     
    File "/home/user/anaconda3/envs/sc_mmlab/lib/python3.7/site-packages/mmcv/runner/iter_based_runner.py", line 134, in run                                   
    iter_runner(iter_loaders[i], **kwargs)                                                                                                                     
    File "/home/user/anaconda3/envs/sc_mmlab/lib/python3.7/site-packages/mmcv/runner/iter_based_runner.py", line 67, in train                                  
    self.call_hook('after_train_iter')                                                                                                                         
    File "/home/user/anaconda3/envs/sc_mmlab/lib/python3.7/site-packages/mmcv/runner/base_runner.py", line 307, in call_hook                                   
    getattr(hook, fn_name)(self)                                                                                                                               
    File "/home/user/anaconda3/envs/sc_mmlab/lib/python3.7/site-packages/mmcv/runner/hooks/evaluation.py", line 262, in after_train_iter                       
    self._do_evaluate(runner)                                                                                                                                  
    File "/home/user/sun_chen/Projects/FsDefect/Main/mmfewshot/mmfewshot/detection/core/evaluation/eval_hooks.py", line 123, in _do_evaluate                   
    self._save_ckpt(runner, key_score)                                                                                                                       
    File "/home/user/anaconda3/envs/sc_mmlab/lib/python3.7/site-packages/mmcv/runner/hooks/evaluation.py", line 330, in _save_ckpt                                 
    if self.compare_func(key_score, best_score):                                                                                                             
    File "/home/user/anaconda3/envs/sc_mmlab/lib/python3.7/site-packages/mmcv/runner/hooks/evaluation.py", line 77, in <lambda>                                
    rule_map = {'greater': lambda x, y: x > y, 'less': lambda x, y: x < y}                                                                                   
TypeError: '>' not supported between instances of 'NoneType' and 'float'     

Bug fix

I noticed that codes in mmcv have already prevented this scenario from happening. BUT there features haven't been updated in mmfewshot and other mmlab repo. So small change following mmcv will save this. turn https://github.com/open-mmlab/mmfewshot/blob/38afc013faa15ee01a9e89b1de81630d5877e752/mmfewshot/detection/core/evaluation/eval_hooks.py#L51 and https://github.com/open-mmlab/mmfewshot/blob/38afc013faa15ee01a9e89b1de81630d5877e752/mmfewshot/detection/core/evaluation/eval_hooks.py#L124

into

if self.save_best and key_score:

[BIGWangYuDong]

Thanks for your reply, this error will fix ASAP. Also, PR is welcome to fix this error