Cannot reproduce VBench-I2V results on CogVideoX-5b-I2V

[Cuogeihong]

We highly appreciate your continuous contribution in t2v, i2v benchmark

Following the guidance, we install the specified environment (cuda 12.1, transformers 4.33.2, torch 2.5.1). We download cogvideox-5b-i2v results from google drive. However, the reproduced results are different from vbench leaderboard. e.g. i2v subject 0.9738 v.s. 0.9719, and we cannot find the explanation. (setting: resolution 3-2, imaging_quality_preprocessing_mode longer)

here is the result json:

results_2025-04-27-19_36_33_eval_results.json

[Jacky-hate]

Could you provide a list of your conda environments？ @Cuogeihong

[Cuogeihong]

We downgraded pytorch to 2.0.1 and solve most inconsistency, but we cannot reproduce camera motion's score: 67.86% v.s. 67.68%

pip environment: pip_list.txt

our camera motion evaluate result: results_2025-04-28-19_45_08_eval_results.json

[Jacky-hate]

We used the pip list you provided and still got the same results as shown on the leaderboard. Could you please share the exact command you used, as well as your conda list (which includes more detailed dependencies)

[Cuogeihong]

We change pytorch to 2.1.1 and camera motion score: 67.65 v.s. 67.68%, still having a small gap

exec command: python evaluate_i2v.py --videos_path $VIDEO_PATH --dimension camera_motion --ratio 3-2 --output_path ./evaluations/$OUTPUT_NAME --full_json_dir ./vbench2_beta_i2v/vbench2_i2v_full_info.json

conda list: conda_list.txt

new camera motion result: results_2025-05-06-18_26_32_eval_results.json