facebookresearch
Significant Gap in Reproduced CoTracker3 Results with `baseline_offline.pth`
Dear authors,
Thank you for sharing your code and pre-trained checkpoints! While reproducing the results reported in your paper, I noticed a significant discrepancy in the performance of CoTracker3 when using the provided checkpoint baseline_offline.pth. Below is a comparison specifically for the baseline checkpoint, as well as the results for other settings.
Results Comparison: baseline_offline.pth
| Validation Dataset | Protocol | Model | Mode | AJ | Delta | OA | |
|---|---|---|---|---|---|---|---|
| Paper: CoTracker3 | DAVIS | First | Base | Offline | 63.3 | 76.2 | 88.0 |
| Reproduced: CoTracker3 | DAVIS | First | Base | Offline | 51.2 | 71.1 | 73.0 |
Results Comparison: Other Settings
| Validation Dataset | Protocol | Model | Mode | AJ | Delta | OA | |
|---|---|---|---|---|---|---|---|
| Paper: CoTracker3 | DAVIS | First | Scaled | Offline | 64.4 | 76.9 | 91.2 |
| Reproduced: CoTracker3 | DAVIS | First | Scaled | Offline | 64.0 | 76.4 | 90.9 |
| Paper: CoTracker3 | DAVIS | First | Base | Online | 64.5 | 76.7 | 89.7 |
| Reproduced: CoTracker3 | DAVIS | First | Base | Online | 62.8 | 76.4 | 88.4 |
| Paper: CoTracker3 | DAVIS | First | Scaled | Online | 63.8 | 76.3 | 90.2 |
| Reproduced: CoTracker3 | DAVIS | First | Scaled | Online | 64.4 | 77.1 | 90.9 |
Questions and Request
- Could you confirm whether the
baseline_offline.pthcheckpoint provided in the repository is the same as the one used in the paper? - Is it possible that an incorrect or incomplete version of the checkpoint was uploaded? If so, could you kindly update the checkpoint to the correct one?
- I used the default configuration from the repository with no modifications.
Looking forward to your response. Thank you for your time and help!
To add a question that is related to reproducibility. The results that I am getting don't seem deterministic, as I am getting different results in different runs. Is there any source of non-determinism in your pipeline? I have tried setting a seed for PyTorch manually, but it does not seem to fix it.
Follow-up on issue regarding baseline_offline.pth
Hi @nikitakaraevv,
I hope you're doing well. I recently opened this issue regarding the baseline_offline.pth file, as I was unable to reproduce the results reported in the paper using the provided checkpoint. I wanted to check if you could kindly verify the checkpoint and update it if necessary.
Please let me know if you need any additional details or logs from my side to assist in the process.
Thank you for your time and effort in maintaining this work!