question about evaluation

[liwenssss]

Hi, thanks for your nice work! I have some questions about the evaluation step. I run the evaluation script:

CFG="grab_rnn"
CFG_CLASS=grab_act_classifier
GPU_IDX=0
TH=0.015
python eval_vae_act_stats_muti_seed.py --iter 500 --nk 10 --bs 5 --num_samp 50 --num_seed 1 --stop_fn 5 --cfg $CFG --cfg_classifier $CFG_CLASS --gpu_index $GPU_IDX --threshold $TH

and get the output like this:

load data from ./data/grab_100_501_wact_candi_test.npz
>>> total params: 1.16M
loading model from checkpoint: ./results/grab_rnn/models/vae_0500.p
>>> total params: 0.21M
loading model from checkpoint: ./results/grab_act_classifier/models/vae_0500.p
OMP: Info #276: omp_set_nested routine deprecated, please use omp_set_max_active_levels instead.
>>>> action pass time used 73.046
>>>> action lift time used 68.177
>>>> action inspect time used 67.383
>>>> action drink time used 68.404
epo 500 mode test threshold 0.015 action classifier grab_act_classifier accuracy: 0.930, test fid 33.419, train fid 44.357, smoothness gt 0.047, smoothness 0.052, div at 100frame 0.000

They do not seem to correspond to the values in the paper. SO how can I get the result of table 2~4 in the paper?

[liwenssss]

and how to calculate the value after ± ?

[wei-mao-2019]

Hi, you need set num_seed to a number greater than 1. In our results we set it to be 5. It means using different number of random seed. The code will save the results for each run and the mean and standard deviation to a csv file to the corresponding logger folder.

The results may vary a little bit due to the randomness.

[liwenssss]

Hi, you need set num_seed to a number greater than 1. In our results we set it to be 5. It means using different number of random seed. The code will save the results for each run and the mean and standard deviation to a csv file to the corresponding logger folder.

The results may vary a little bit due to the randomness.

Sorry, it's my first time in this direction and I don't understand some parts, such as which part of the ADE indicator in Table 3 corresponds to the test results