Example for recognising on an open-set of tags

[mrdbourke]

Hi there,

Firstly, fantastic work and thank you for sharing!

Second, do you mind providing a small code example to recognize an open-set of tags?

Eg.

from ram import get_transform, inference_ram, inference_tag2text
from ram.models import ram, tag2text_caption

ram_model = ram(pretrained=ram_checkpoint, image_size=image_size, vit='swin_l').eval().to(device)

# Setup image
image = ... 

# Custom tags
tags = ["house", "car", "pig"] 

# Perform inference
result = inference_ram(model, image, custom_tags)

[xinyu1205]

Hi, for recognize open-set of tags, you can refer to "RAM Inference on Unseen Categories (Open-Set)" in README.md. And thank you for providing such a concise and amazing code reference, and we will consider implementing this in the feature. Best regards!

[majinyu666]

Hi there,

Firstly, fantastic work and thank you for sharing!

Second, do you mind providing a small code example to recognize an open-set of tags?

Eg.

from ram import get_transform, inference_ram, inference_tag2text
from ram.models import ram, tag2text_caption

ram_model = ram(pretrained=ram_checkpoint, image_size=image_size, vit='swin_l').eval().to(device)

# Setup image
image = ... 

# Custom tags
tags = ["house", "car", "pig"] 

# Perform inference
result = inference_ram(model, image, custom_tags)

Hi, with PR #63 now you can set up custom dataset, whether for open-set or not. Please refer to the "Batch Inference and Evaluation" section of readme, and set up dataset following datasets/openimages-rare-200.

Here are main steps of setting up datasets:

1. mkdir datasets/my_dataset/.

2. ln -s /path/to/image_folder datasets/my_dataset/imgs.

3. Create tag list datasets/my_dataset/my_dataset_ram_taglist.txt. This is where you put any custom tags. (But for closed-set, these tags should be subset of the label system, i.e., subset of ram/data/tam_tag_list.txt.)

   my_tag_1
   my_tag_2
   my_tag_3
   

4. Create image list datasets/my_dataset/my_dataset_ram_annots.txt. Ignore annots if you don't care about metrics.

   subfolder1/subfolder2/aaa.jpg
   subfolder1/subfolder2/bbb.jpg,my_tag_2,my_tag_3
   

[SeeknnDestroy]

hi @majinyu666, @xinyu1205,

I want to create my custom dataset for tag2text batch inference. But I saw that it has different format, tagidlist.txt and idannots.txt. Can you please show an example for each, what they refer to?