textgenrnn icon indicating copy to clipboard operation
textgenrnn copied to clipboard

Published 20 hours ago • verified publisher icon minimaxir

random keyerrors

Open ihavetoomanyquestions opened this issue 6 years ago • 4 comments

Hi,

When I train my models, I randomly get the following key error. Any idea what might be causing the problem?

Traceback (most recent call last): File "train.py", line 39, in word_level=cfg['model_config']['word_level']) File "textgenrnn/textgenrnn.py", line 262, in train_new_model **kwargs) File "textgenrnn/textgenrnn.py", line 195, in train_on_texts validation_steps=val_steps File "miniconda2/envs/py36/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper return func(*args, **kwargs) File "miniconda2/envs/py36/lib/python3.6/site-packages/keras/engine/training.py", line 1426, in fit_generator initial_epoch=initial_epoch) File "miniconda2/envs/py36/lib/python3.6/site-packages/keras/engine/training_generator.py", line 229, in fit_generator callbacks.on_epoch_end(epoch, epoch_logs) File "miniconda2/envs/py36/lib/python3.6/site-packages/keras/callbacks.py", line 77, in on_epoch_end callback.on_epoch_end(epoch, logs) File "textgenrnn-s/textgenrnn/utils.py", line 174, in on_epoch_end max_gen_length=self.max_gen_length) File "textgenrnn-s/textgenrnn/textgenrnn.py", line 94, in generate_samples self.generate(n, temperature=temperature, **kwargs) File "textgenrnn-s/textgenrnn/textgenrnn.py", line 83, in generate max_gen_length) File "textgenrnn-s/textgenrnn/utils.py", line 69, in textgenrnn_generate next_char = indices_char[next_index] KeyError: 0

ihavetoomanyquestions avatar Jun 09 '18 02:06 ihavetoomanyquestions

What's happening is that the model is trying to select a value representing an unknown char/word. I thought I put safeguards against this but apparently that was insufficient.

minimaxir avatar Jun 10 '18 22:06 minimaxir

is there a quick fix I can do to the code, or is it more complicated? (would a try/catch somewhere at least prevent the script to crash?) because no matter what i do, retrain/use another data etc. it always come to that point where it crashes :(

ihavetoomanyquestions avatar Jun 10 '18 23:06 ihavetoomanyquestions

I am getting this as well. I am using a cleaned up, shorter version of the cornell movie dialog corpus.

Adattilio avatar Feb 25 '19 23:02 Adattilio

@minimaxir the problem is still there, any ETC on this issue?

nm17 avatar Nov 27 '19 15:11 nm17