Transformer pre-training

[Z-Yh-June]

hello,your work is amazing! in this paper ,i have a problem.Transformer can get better results only after pre-training in a large data set. I don't know whether you have pre-trained your Transformer frame and fine-tuned it or just loaded resnet model parameters.