VBench icon indicating copy to clipboard operation
VBench copied to clipboard

zero output

Open DidiD1 opened this issue 1 year ago • 5 comments

I want to use this to evaluate my own videos, but it's strange that the output is all 0. Could you help me to correctly use the metricks to eval, thanks!

DidiD1 avatar Dec 03 '24 03:12 DidiD1

the code i use:

python evaluate.py \
 --videos_path ./VBench/11_29_3s \
 --dimension "motion_smoothness" \
 --mode "custom_input"

and the output be like: { "subject_consistency": [ 0.0, [ { "video_path": xxx, "video_results": 0.0 }, { "video_path":xxx, "video_results": 0.0 }, { "video_path": xxx, "video_results": 0.0 },

DidiD1 avatar Dec 03 '24 03:12 DidiD1

same problem here https://github.com/Vchitect/VBench/issues/87

rahimentezari avatar Dec 04 '24 11:12 rahimentezari

same problem here #87 I track the bug found a strange thing is that, the image_features of the clip is all 0, which leads to the result of compute_background_consistency be zero. I don't know what's the problem, maybe the version of package?

DidiD1 avatar Dec 05 '24 02:12 DidiD1

Can you provide ls -lh ~/.cache/vbench/* and version here? Seems like the problem is not specific to CLIP as subject_consistency does not use it (https://github.com/Vchitect/VBench/issues/87).

NattapolChan avatar Dec 05 '24 12:12 NattapolChan

same problem here #87 I track the bug found a strange thing is that, the image_features of the clip is all 0, which leads to the result of compute_background_consistency be zero. I don't know what's the problem, maybe the version of package?

I encountered the same problem and found the same thing as you. Besides, I noticed that the first model inference process took a unusually long time(approximately 10-12minutes) while other steps could finish in just minutes. Have you solved this bug?

zhongyu-zhao avatar Jan 06 '25 06:01 zhongyu-zhao