When evaluating using gen_eval.py it throws error

[kirk86]

Traceback (most recent call last):
  File "gen_eval.py", line 35, in <module>
    model_list = [torch.load(file).to(device) for file in model_path.file_dict.values()]
  File "gen_eval.py", line 35, in <listcomp>
    model_list = [torch.load(file).to(device) for file in model_path.file_dict.values()]
AttributeError: 'collections.OrderedDict' object has no attribute 'to'

As the message says you're trying to send an ordered dict to device, while I believe it should be the actual model. I tried loading the ordered dict to the model_path.base_model which is a resnet18 but there's mismatch. Where should that ordered dict go?

[AlexMeinke]

Hey Kirk, thanks for letting me know. Indeed there is a problem. The training code itself saves the actual models (so that's what the gen_eval.py script wants) but for portability I hosted pretrained weights instead of models.

Loading a CCU model from weights requires a few steps since it consists of more parts than just the base model so I just added a script to hopefully make this more convenient.

https://github.com/AlexMeinke/certified-certain-uncertainty/commit/8f33cd5935f633acdb1c40fbb5d592d4b4440b51

Thanks again for pointing out the issue and let me know if there are more problems.

[kirk86]

Thanks Alex, there are a couple of other issues (e.g. missing arguments or fixed paths) which might not show up when you execute gen_eval.py on your local computer because you have those paths fixed on your local pc. I'd suggest running gen_eval.py on colab to discover the uncaught errors which might not show on your local pc.

[AlexMeinke]

Well, I mean, yea. Paths need to be configured manually by the user. Specifically utils/dataloaders.py and model_paths.py I should probably point to these locations explicitly in the Readme. Thanks a lot for pointing that out