Question about evaluation on BADJA

[cyrala]

Hi ! I ran test_on_badja.py and the average PCK-T of pips is 58.7(nvidia 3090), while I founded that in the function of run_pips, the calculation of pck includes the accs of the initial labeled frame(pck =100). But in the function of run_raft, it doesn't include accs of the initial labeled frame, which will lead to the higher pck of pips. I think this processing method is not reasonable. Can you tell me why? And when I calculate pck of pips according to run_raft, I get a result of 53.0(nvidia 3090) Next, I founded that the test result of the pips' PCK are different on different GPUs(53.0 on nvidia 3090, 51.5 on nvidia 1080), but I don't know why.

[aharley]

The first issue there sounds like a bug. Good catch! I will try to fix this soon, and update the results.

Separate from this, however, it seems like you're getting results a few points lower than the paper (even with the bug included). Is it possible that something else is different in your setup?

For the second issue (differences across GPUs) -- I don't know.

[aharley]

@Rosie-dala I think I fixed the bug in the badja test script.

Check it out: https://github.com/aharley/pips/blob/main/test_on_badja.py#L142-L155