selfmonitoring-agent
selfmonitoring-agent copied to clipboard
Published
20 hours ago •
chihyaoma
chihyaoma
Performance Gap (significantly lower than reported)
Hi,
I have tried to reproduce the reported result. However, my results are lower than the paper claimed. The results are shown below:
Evaluating on val_seen env ...
Epoch: [205][1/16] Time 44.433 (44.433) Loss 0.0000 (0.0000)
Epoch: [205][2/16] Time 41.156 (42.794) Loss 0.0000 (0.0000)
Epoch: [205][3/16] Time 40.692 (42.093) Loss 0.0000 (0.0000)
Epoch: [205][4/16] Time 49.232 (43.878) Loss 0.0000 (0.0000)
Epoch: [205][5/16] Time 56.059 (46.314) Loss 0.0000 (0.0000)
Epoch: [205][6/16] Time 50.356 (46.988) Loss 0.0000 (0.0000)
Epoch: [205][7/16] Time 38.051 (45.711) Loss 0.0000 (0.0000)
Epoch: [205][8/16] Time 38.715 (44.837) Loss 0.0000 (0.0000)
Epoch: [205][9/16] Time 37.932 (44.070) Loss 0.0000 (0.0000)
Epoch: [205][10/16] Time 38.539 (43.516) Loss 0.0000 (0.0000)
Epoch: [205][11/16] Time 38.241 (43.037) Loss 0.0000 (0.0000)
Epoch: [205][12/16] Time 47.775 (43.432) Loss 0.0000 (0.0000)
Epoch: [205][13/16] Time 54.741 (44.302) Loss 0.0000 (0.0000)
Epoch: [205][14/16] Time 52.321 (44.875) Loss 0.0000 (0.0000)
Epoch: [205][15/16] Time 38.056 (44.420) Loss 0.0000 (0.0000)
Epoch: [205][16/16] Time 37.984 (44.018) Loss 0.0000 (0.0000)
| nav_error: 3.8490198931233213 | oracle_error: 2.3369494706951466 | steps: 6.107843137254902 | lengths: 10.45375464930404 | spl: 0.6103185273231977 | success_rate: 0.6460784313725491 | oracle_rate: 0.7294117647058823
Evaluating on val_unseen env ...
Epoch: [205][1/37] Time 38.728 (38.728) Loss 0.0000 (0.0000)
Epoch: [205][2/37] Time 36.856 (37.792) Loss 0.0000 (0.0000)
Epoch: [205][3/37] Time 36.631 (37.405) Loss 0.0000 (0.0000)
Epoch: [205][4/37] Time 37.941 (37.539) Loss 0.0000 (0.0000)
Epoch: [205][5/37] Time 36.848 (37.401) Loss 0.0000 (0.0000)
Epoch: [205][6/37] Time 36.870 (37.312) Loss 0.0000 (0.0000)
Epoch: [205][7/37] Time 37.395 (37.324) Loss 0.0000 (0.0000)
Epoch: [205][8/37] Time 36.739 (37.251) Loss 0.0000 (0.0000)
Epoch: [205][9/37] Time 37.376 (37.265) Loss 0.0000 (0.0000)
Epoch: [205][10/37] Time 36.645 (37.203) Loss 0.0000 (0.0000)
Epoch: [205][11/37] Time 36.693 (37.156) Loss 0.0000 (0.0000)
Epoch: [205][12/37] Time 37.355 (37.173) Loss 0.0000 (0.0000)
Epoch: [205][13/37] Time 36.837 (37.147) Loss 0.0000 (0.0000)
Epoch: [205][14/37] Time 37.657 (37.184) Loss 0.0000 (0.0000)
Epoch: [205][15/37] Time 37.261 (37.189) Loss 0.0000 (0.0000)
Epoch: [205][16/37] Time 36.708 (37.159) Loss 0.0000 (0.0000)
Epoch: [205][17/37] Time 36.177 (37.101) Loss 0.0000 (0.0000)
Epoch: [205][18/37] Time 37.224 (37.108) Loss 0.0000 (0.0000)
Epoch: [205][19/37] Time 36.919 (37.098) Loss 0.0000 (0.0000)
Epoch: [205][20/37] Time 36.471 (37.067) Loss 0.0000 (0.0000)
Epoch: [205][21/37] Time 37.841 (37.103) Loss 0.0000 (0.0000)
Epoch: [205][22/37] Time 36.392 (37.071) Loss 0.0000 (0.0000)
Epoch: [205][23/37] Time 37.118 (37.073) Loss 0.0000 (0.0000)
Epoch: [205][24/37] Time 37.091 (37.074) Loss 0.0000 (0.0000)
Epoch: [205][25/37] Time 36.840 (37.064) Loss 0.0000 (0.0000)
Epoch: [205][26/37] Time 36.788 (37.054) Loss 0.0000 (0.0000)
Epoch: [205][27/37] Time 37.286 (37.062) Loss 0.0000 (0.0000)
Epoch: [205][28/37] Time 36.880 (37.056) Loss 0.0000 (0.0000)
Epoch: [205][29/37] Time 37.105 (37.058) Loss 0.0000 (0.0000)
Epoch: [205][30/37] Time 37.046 (37.057) Loss 0.0000 (0.0000)
Epoch: [205][31/37] Time 43.863 (37.277) Loss 0.0000 (0.0000)
Epoch: [205][32/37] Time 44.296 (37.496) Loss 0.0000 (0.0000)
Epoch: [205][33/37] Time 38.315 (37.521) Loss 0.0000 (0.0000)
Epoch: [205][34/37] Time 38.055 (37.537) Loss 0.0000 (0.0000)
Epoch: [205][35/37] Time 37.899 (37.547) Loss 0.0000 (0.0000)
Epoch: [205][36/37] Time 47.541 (37.825) Loss 0.0000 (0.0000)
Epoch: [205][37/37] Time 54.212 (38.267) Loss 0.0000 (0.0000)
| nav_error: 6.141163837885611 | oracle_error: 3.6298467382790323 | steps: 6.171988080034057 | lengths: 10.225189720171494 | spl: 0.3966670362847731 | success_rate: 0.44785014899957426 | oracle_rate: 0.5798212005108557
In the paper, Table 2 (row number 7), the expected result should be
val_seen (NE | SR | OSR | SPL): 3.23 | 0.70 | 0.78 | 0.66 val_unseen(NE | SR | OSR | SPL): 5.04 | 0.57 | 0.70 | 0.51. However, I obtained val_seen SPL 0.61, 5% lower and val_unseen SPL 0.396, 12% lower.
My configurations are posted as follows:
# Name Version Build Channel
python 3.8.2 hcf32534_0
pytorch 1.4.0 py3.8_cuda10.1.243_cudnn7.6.3_0 pytorch
numpy 1.18.1 py38h4f9e942_0
networkx 2.4 pypi_0 pypi
torchvision 0.5.0 py38_cu101 pytorch
I am facing the same problem here yielding almost same perforance with you. Have you found any clue?
I am facing the same problem here yielding almost same perforance with you. Have you found any clue?
I still do not have any clue. If you find the answer, can you share with us here ? This is really wired.
Same here
If you find the answer, can you share with us here ? This is really wired.