open vocabulary object detection in the wild

[meidachen]

trafficstars

First, thanks a lot for the great work! Congratulations on your achievements on different challenges! I'm trying to use your pre-trained model to do open vocabulary object detection in the wild kind of task, by simply providing an image and giving some arbitrary vocabulary (e.g., house, car, tree, light pole, fence, etc.) and hopping to do an object detection on the given image. I'm unsure if I understand your work's capability correctly and if this is a suitable task. I tried to naively change the following code in the demo script to hard code the object classes. https://github.com/MasterBin-IIAU/UNINEXT/blob/5ab03865d05d61c23799ae0bd3305b160ceb3f34/projects/UNINEXT/predictor.py#L281

to:

        test_categories = [{'color': [220, 20, 60], 'isthing': 1, 'id': 1, 'name': 'ground'}, {'color': [119, 11, 32], 'isthing': 1, 'id': 2, 'name': 'house'}, {'color': [0, 0, 142], 'isthing': 1, 'id': 3, 'name': 'tree'}, {'color': [0, 0, 230], 'isthing': 1, 'id': 4, 'name': 'car'}, {'color': [106, 0, 228], 'isthing': 1, 'id': 5, 'name': 'light pole'},{'color': [106, 0, 228], 'isthing': 1, 'id': 6, 'name': 'fence'}]
        # test_categories = None
        if test_categories is not None:
            prompt_test, positive_map_label_to_token = create_queries_and_maps(test_categories, self.tokenizer) # for example, test_categories = [{"name": "person"}]
            self.prompt_test_dict["xxx"] = prompt_test
            self.positive_map_label_to_token_dict["xxx"] = positive_map_label_to_token

But when testing on different images, it seems like only the cars can be detected and the other objects can not. Could you please help me with this?

Thank you in advance!