Non-Reproducible / Weird Uncertainty Results

[FloLins]

Hello,

I wanted to check the uncertainty properties of my SWAG models and can not reproduce the values from the paper. When I train a VGG16/CIFAR10 model with the parameters from the paper and use your uncertainty script like this:

python uncertainty.py --data_path= --model=VGG16 --dataset=CIFAR10 --method=SWAG --scale=0.5 --use_test --cov_mat --file=<path/swag-300.pt> --save_path=<save_path>

It will result in a good accuracy, but in a huge NLL:

30/30 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 79/79 [00:00<00:00, 88.94it/s] Accuracy: 0.9366 NLL: 1990.9390262566017

Thanks in advance

[wjmaddox]

i'm pretty sure that's close to the correct number but off by a scaling factor equal to the size of the dataset, https://github.com/wjmaddox/swa_gaussian/blob/61a918fa1c3e732dd19fc278ad3a5969f24d5a72/experiments/uncertainty/uncertainty.py#L214

scaling factor is 10k, which produces a nll of 0.199, which is what's in fig 2 of the paper