RuntimeError("The program has been running off.")

[kizunasunhy]

In Epoch:16/Step:17296, got best dev performance: Seq2SeqSpanMetric: f=0.0, rec=0.0, pre=0.0, em=0.1836 .... File "/root/work/BARTNER/model/callbacks.py", line 107, in on_valid_end raise RuntimeError("The program has been running off.") RuntimeError: The program has been running off.

Does this mean the model stops by itself because of convergence failure? What can I do now? To train it again? Thanks

[yhcc]

Just train it again, it is because of the convergence failure.

[summer-la]

Did the results converge after re-running? @kizunasunhy I'm having the same issue now.