Huge accuracy drop from AR with 1 refinement to NAR with 2 refinements

[killawhale2]

I was testing the model using custom STR data and found that there is a huge accuracy gap (up to 20+% word accuracy) going from AR with 1 refinement to NAR with 2 refinements. This is strange as the results in the paper shows only 1~2% word accuracy gap between the two decoding schemes. The results with AR with 1 refinement are looking very good, which puzzles me as the training resulted in one effective decoding scheme. Any help or anecdotal information on this would be very much appreciated!

[baudm]

Are you using the pretrained weights as is? What's the difference between the NED of AR vs NAR? For the standard STR benchmarks, the difference between the two is within 2%. For the harder datasets (COCO, Uber), the difference is a bit wider. Care to share your custom STR data? It's hard to speculate about something that I cannot directly test.