MonoDETR icon indicating copy to clipboard operation
MonoDETR copied to clipboard

Training results fluctuate significantly

Open Z1740220020 opened this issue 2 years ago • 8 comments

Thank you so much for doing such a great job and providing great code. I'm having problems reproducing your results. I used 4 3090 graphics cards, and the settings of parameters such as batchsize and learning rate are all set by default. I find that AP always fluctuates between 17 and 19. The fluctuations are large. Is this normal?

Z1740220020 avatar Sep 07 '22 01:09 Z1740220020

Same question, but i got more higher score in 3D AP R40: 27.8836, 19.3681, 15.9685(test set) with default parameters.

SibylGao avatar Sep 09 '22 07:09 SibylGao

Same question . I run many times but the highest score is around19.0

pummi823 avatar Sep 11 '22 08:09 pummi823

Same question, but i got more higher score in 3D AP R40: 27.8836, 19.3681, 15.9685(test set) with default parameters.

May I ask what kind of hardware equipment you train on? I was wondering if it could be due to the fluctuations caused by multi-GPU training. The author mentioned in the paper that he only used a single 3090. And I used four 3090 graphics cards and the batch size and learning rate were not adjusted (the same as in the paper).

Z1740220020 avatar Sep 11 '22 08:09 Z1740220020

Same question . I run many times but the highest score is around19.0

May I ask how many graphics cards you used for training?

Z1740220020 avatar Sep 11 '22 09:09 Z1740220020

Same question . I run many times but the highest score is around19.0

May I ask how many graphics cards you used for training?

Just one

pummi823 avatar Sep 13 '22 02:09 pummi823

Same question, but i got more higher score in 3D AP R40: 27.8836, 19.3681, 15.9685(test set) with default parameters.

May I ask what kind of hardware equipment you train on? I was wondering if it could be due to the fluctuations caused by multi-GPU training. The author mentioned in the paper that he only used a single 3090. And I used four 3090 graphics cards and the batch size and learning rate were not adjusted (the same as in the paper).

I use 4 v100 graphics cards.

SibylGao avatar Sep 13 '22 05:09 SibylGao

Thank you so much for doing such a great job and providing great code. I'm having problems reproducing your results. I used 4 3090 graphics cards, and the settings of parameters such as batchsize and learning rate are all set by default. I find that AP always fluctuates between 17 and 19. The fluctuations are large. Is this normal?

Hi, I am wondering whether the batchsize will influence the results, i made it as 4 (not 16), but got much worse AP. Did you guys continue trying to rise up the AP

Shinkair avatar Feb 28 '23 05:02 Shinkair

Thank you so much for doing such a great job and providing great code. I'm having problems reproducing your results. I used 4 3090 graphics cards, and the settings of parameters such as batchsize and learning rate are all set by default. I find that AP always fluctuates between 17 and 19. The fluctuations are large. Is this normal?

Hi, I am wondering whether the batchsize will influence the results, i made it as 4 (not 16), but got much worse AP. Did you guys continue trying to rise up the AP

have you solved this issue? I have the same problem.

tranquan687 avatar Nov 23 '23 03:11 tranquan687