Should Dice coefficient for empty reference and empty test be 1?

[baumgach]

Hi there, thanks for making the code public!

My question is in reference to the following line:

https://github.com/MIC-DKFZ/nnUNet/blob/6d02b5a4e2a7eae14361cde9599bbf4ccde2cd37/nnunet/evaluation/metrics.py#L118

Shouldn't an empty reference set and an empty test set result in a Dice coefficient of 1 rather than 0? I.e. if no pixels of a class are present in the image and the network predicts no mask, then the prediction is correct. Or am I missing something here? (The same would go for the Jaccard score.)

Cheers, Christian

[FabianIsensee]

Maybe. This is up for debate. It's a 0/0 situation and thus undefined. Most people set it to 1 in that case but I don't like how this biases the results towards tuning for empty predictions. This doesn't matter in nnU-Net though because we use nan_for_nonexisting=True and aggregate with np.nanmean meaning that 0/0 cases are ignored in aggregation. It's conservative but in my eyes the correct approach when optimizing average scores (it's a different story when using ranke-then-aggregate)

[sten2lu]

Hi @baumgach ,

Since this issue has already been discussed and there has been no recent activity, we will close this issue in 2 weeks (Monday 14th of August). If you would like to add to your argument, please answer here so that we may find an improved solution.

Best, Carsten