Wanjun

I'm doing the sentence tagging task. I repeat my experiment with same training and testing data, and exactly same code, just repeat. However, get different predictions every time. The testing...

请问大家在bert训练时遇到过显存泄漏吗？

3

前面几千个step比较稳定，占用一半的显存，训练正常，loss会下降，后面会突然持续增大显存占用，然后爆掉，基本上报错地方都在这里。 求解~~ File "/home/anaconda3/envs/python2.7/lib/python2.7/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/home/embed.py", line 84, in forward text_encoded_layer, _ = self.bert_model(text_var, text_segments_ids, output_all_encoded_layers=False) File "/home/anaconda3/envs/python2.7/lib/python2.7/site-packages/torch/nn/modules/module.py", line 532, in __call__...