RCIL
RCIL copied to clipboard
Reproduced results lower than the reported ones
Hi, I directly ran the released codes without any modification. However, I found that the obtained results are lower than the reported ones by >1 percent point, especially the 10-1 setting with a large gap on the base (0-10) classes.
Relevant log files are provided for your reference. Could you advise the possible reasons that may cause such a problem? Thanks a lot.
15-5 | 15-1 | 10-1 | |||||||
---|---|---|---|---|---|---|---|---|---|
0-15 | 16-20 | all | 0-15 | 16-20 | all | 0-10 | 11-20 | all | |
Reported | 78.8 | 52.0 | 72.4 | 70.6 | 23.7 | 59.4 | 55.4 | 15.1 | 34.3 |
Reproduced | 76.7 | 48.4 | 70.0 | 69.0 | 20.5 | 57.4 | 38.0 | 13.4 | 26.3 |
Hi, @Ze-Yang , I upload the experimental results for each step at: experimental results, and I will update the files of 10-10s soon, when I can access the server.
Here are some results in my git history:
-
15-5s setting:
-
15-5 setting:
My environment is requirements. I re-train the 15-5s overlapped before, and find that the performance of step-0 plays a key role for the final performance. I will re-train other settings when I can access the server.
Hi, I am still not able to reproduce especially the 10-1 setting results. What I obtain is far lower than the reported number. Meanwhile, I observe a strange phenomenon that the background performance drops drastically at step 8 and become 0 at step 9. Do you have a similar problem? I attach the results for all steps for your reference. Thanks.
@Ze-Yang I'm sorry for the late reply. We have fixed bugs in the repository, and you can find the re-implemented results on VOC and ADE20K. If there is any questions, please feel easy to let us know. Thanks for your help.