VPE
VPE copied to clipboard
mibastro
Confusion Regarding belga2toplogo result
I got the following result in the file best_iou.txt for belga2toplogo (main_train_val_test.py) which doesn't seem to match the reported accuracy in the paper: Best score epoch: 100 acc cls: 0.6151 acc all: 0.5991 rank mean: 66.5124 acc tr cls: 0.5575 acc te cls: 0.6632 top3: 0.7547 top5: 0.8644
Am I looking at the correct file?
best_iou.txt file records the best performance during training. The numbers reported in the paper is using the validation set. This means performance in the paper is the test accuracy of a model that tops performance in the validation set. So it is natural that best_iou.txt shows higher numbers than best_iou_val.txt
This is the entry in best_iou_val.txt for belga2toplogo Best score epoch: 420 acc cls: 0.6022 acc all: 0.6072 rank sample mean: 0.8964 rank cls mean: 0.8940 acc tr cls: 0.8399 acc te cls: 0.5683 top3: 0.7262 top5: 0.7829
The entry in best_iou_triggeredByVal.txt for belga2toplogo Best score epoch: 420 acc cls: 0.4306 acc all: 0.4057 rank mean: 62.7107 acc tr cls: 0.5040 acc te cls: 0.3694 top3: 0.6922 top5: 0.8219
Doesnot match the reported result. Is this the result you are referring to?
Yes, best_iou_triggeredByVal.txt is what I meant. (Not best_iou_val.txt) I will reproduce the model and upload it here later.
You can tune the training process by lowering 'prototype_sampling_rate' parameter in the belga2toplogo.py. (or set it to zero.) This is hyperparameter and tunning this parameter changes performance.
The model is trainable without using prototypes as inputs. Using prototypes in the training process can be optional.
With prototype_sampling_rate = 0 the entry in best_iou_triggeredByVal.txt for belga2toplogo
Best score epoch: 512 acc cls: 0.5210 acc all: 0.5024 rank mean: 62.6116 acc tr cls: 0.6265 acc te cls: 0.4331 top3: 0.7040 top5: 0.7948
Still not close to the reported results.
I have an important deadline soon, I will get back to you later. 'Belga2toplogo' experiment performance fluctuates compared to other experiments. This might be due to the dataset domain gap which is larger compared to other experiment scenarios. The lower performance compared to the paper might be due to long training epochs and overfitting. If you set smaller epochs you will get better result (ex: 200 epochs)