VRepair
VRepair copied to clipboard
SteveKommrusch
Replicating the target domain training results
Hello @chenzimin, Can you help me take a look at this log? Somehow my results showed better results than yours using a smaller architecture with half the size? Here is my log: log file
I can see the log with the new link. The training accuracy is really high (almost 100%), but the validation accuracy is only [2022-05-06 01:14:23,334 INFO] Validation accuracy: 49.948. So it seems like it is overfitting to the training data.
This log is the training log for target domain only, isn't it too good to be true comparing to the results you reported in your paper ?
On Fri, May 13, 2022, 7:33 PM chenzimin @.***> wrote:
I can see the log with the new link. The training accuracy is really high (almost 100%), but the validation accuracy is only [2022-05-06 01:14:23,334 INFO] Validation accuracy: 49.948. So it seems like it is overfitting to the training data.
— Reply to this email directly, view it on GitHub https://github.com/SteveKommrusch/VRepair/issues/10#issuecomment-1126008179, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQJ6JV4HOBGX2NXOHBNW6WDVJZDZXANCNFSM5VIWLRBA . You are receiving this because you modified the open/close state.Message ID: @.***>
I believe that OpenNMT-py reports per token accuracy, whereas we calculate the sequence accuracy.
Do you have the code for calculating sequence level accuracy cause I can not seem to find it in the repo or do I have to reimplement the code myself?
On Sat, May 14, 2022 at 2:44 PM chenzimin @.***> wrote:
I believe that OpenNMT-py reports per token accuracy, whereas we calculate the sequence accuracy.
— Reply to this email directly, view it on GitHub https://github.com/SteveKommrusch/VRepair/issues/10#issuecomment-1126664642, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQJ6JV6IMPRVGV4ZM5TA6T3VJ5KU7ANCNFSM5VIWLRBA . You are receiving this because you modified the open/close state.Message ID: @.***>
We use the src/compare.py file to compare the output of our model when given test samples with the expected sequence for the sample. For example: "python ../../src/compare.py --src=xlate.txt --tgt=tgt-test.txt -v > pass.txt".
The usage for compare.py is: Usage: python compare.py --src [src_file] --tgt [tgt_file] [-v] the -v option will print out all passing cases Example: python compare.py --src pred-test.txt --tgt tgt-test.txt
One more thing, you can use src/find_best_model_and_translate_config.py to generate command to generate the predictions from the model with the best validation accuracy.