Note on inference instructions.

[Haradai]

Hi, when attempting to inference this model uisng the test_seesr script I ran into a small problem regarding the DAPE checkpoint that I found a solution to.

Initially I followed the md instructions:

1. cloned this repo
2. donwloaded SD from Hugging face and put into preset/models
3. Downloaded SeeSR and DAPE checkpoint and put also into preset/models.

Then I ran into the following error:

Loading default thretholds from .txt....
--------------
preset/models/ram_swin_large_14m.pth
--------------
Traceback (most recent call last):
  File "test_seesr.py", line 268, in <module>
    main(args)
  File "test_seesr.py", line 168, in main
    model = load_tag_model(args, accelerator.device)
  File "test_seesr.py", line 125, in load_tag_model
    model = ram(pretrained='preset/models/ram_swin_large_14m.pth',
  File "/home/jmrocafort/Documents/ImageSR_human_compare_experiment/Models/SeeSR/SeeSR/ram/models/ram_lora.py", line 325, in ram
    model, msg = load_checkpoint_swinlarge(model, pretrained, kwargs)
  File "/home/jmrocafort/Documents/ImageSR_human_compare_experiment/Models/SeeSR/SeeSR/ram/models/utils.py", line 296, in load_checkpoint_swinlarge
    raise RuntimeError('checkpoint url or path is invalid')
RuntimeError: checkpoint url or path is invalid

It cannot find the base ram_swin_large_14m.pth checkpoint. Which seemed weird as DAPE.pth should already be the checkpoint of the finetuned ram in use. So attempted to replace the path with the DAPE.pth -> line 125 of test_seesr.py

def load_tag_model(args, device='cuda'):
    
    model = ram(pretrained='preset/models/DAPE.pth',
                pretrained_condition=args.ram_ft_path,
                image_size=384,
                vit='swin_l')
    model.eval()
    model.to(device)
    
    return model

This won't work as I get the error:

--------------
Traceback (most recent call last):
  File "test_seesr.py", line 268, in <module>
    main(args)
  File "test_seesr.py", line 168, in main
    model = load_tag_model(args, accelerator.device)
  File "test_seesr.py", line 125, in load_tag_model
    model = ram(pretrained='preset/models/DAPE.pth',
  File "/home/jmrocafort/Documents/ImageSR_human_compare_experiment/Models/SeeSR/SeeSR/ram/models/ram_lora.py", line 325, in ram
    model, msg = load_checkpoint_swinlarge(model, pretrained, kwargs)
  File "/home/jmrocafort/Documents/ImageSR_human_compare_experiment/Models/SeeSR/SeeSR/ram/models/utils.py", line 298, in load_checkpoint_swinlarge
    state_dict = checkpoint['model']
KeyError: 'model'

So I'm unsure that DAPE.pth is a whole model but rather only the weights state_dict? ( I'm novice in pytorch so I'm very unsure)

Solution: Just had to download a ram_swin_large_14m checkpoint from hugging face and put it in preset/models. https://huggingface.co/spaces/xinyu1205/recognize-anything/blob/main/ram_swin_large_14m.pth

Finally I've seen from the script logs how the script loads this base model and then loads the finetuned weights from DAPE.pth on it.

Request: I just wanted to comment this so maybe the instructions could be updated to clarify this so other people do not run in the same issue. Hope explaining my whole chain of thought wasn't too long, I thought it might be useful because maybe nothing is missing and I just missed some step. Thank you :)