The correct way to prompt for 0-shot video classification

[Eugleo]

Thanks for your great work! This is more of a question than a bug report. Say I have a short clip (8s) of a door being opened and another one where the door is being closed. What would you say would be a good way to describe these scenes in way that the cosine similarities between the videos and the descriptions could be used to classify the videos as either closing or opening without any additional fine-tuning?

I tried my hand at this but found it's pretty hard to come up with a good prompt. I thought maybe you would have better intuitions for this.

[Andy1621]

How about directly testing their similarity? Besides, your project is like PAXION and you can refer it for some motivation.

[Eugleo]

Good pointer, thanks. I was mostly asking for specific prompting tips, however; I did run a few experiments but it seems I have to be missing some larger pattern — I didn't find a description that would work well consistently.

EDIT: Looking at PAXION, maybe the model just can't tell the difference in the first place.