Internal error

[anguslou]

Training new model w/ 4-layer, 128-cell Bidirectional LSTMs Training on 92,139 character sequences. Epoch 1/10

InternalError Traceback (most recent call last) /usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args) 1321 try: -> 1322 return fn(*args) 1323 except errors.OpError as e:

/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in _run_fn(feed_dict, fetch_list, target_list, options, run_metadata) 1306 return self._call_tf_sessionrun( -> 1307 options, feed_dict, fetch_list, target_list, run_metadata) 1308

/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in _call_tf_sessionrun(self, options, feed_dict, fetch_list, target_list, run_metadata) 1408 self._session, options, feed_dict, fetch_list, target_list, -> 1409 run_metadata) 1410 else:

InternalError: Failed to call ThenRnnForward with model config: [rnn_mode, rnn_input_mode, rnn_direction_mode]: 2, 0, 0 , [num_layers, input_size, num_units, dir_count, seq_length, batch_size]: [1, 100, 128, 1, 40, 1024] [[Node: rnn_1_3/CudnnRNN = CudnnRNN[T=DT_FLOAT, _class=["loc:@training_1/RMSprop/gradients/rnn_1_3/CudnnRNN_grad/CudnnRNNBackprop"], direction="unidirectional", dropout=0, input_mode="linear_input", is_training=true, rnn_mode="lstm", seed=87654321, seed2=0, _device="/job:localhost/replica:0/task:0/device:GPU:0"](rnn_1_3/transpose, rnn_1_3/ExpandDims_1, rnn_1_3/ExpandDims_1, rnn_1_3/concat)]] [[Node: loss_3/mul/_445 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_3927_loss_3/mul", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

During handling of the above exception, another exception occurred:

InternalError Traceback (most recent call last) in () 20 max_length=model_cfg['max_length'], 21 dim_embeddings=model_cfg['dim_embeddings'], ---> 22 word_level=model_cfg['word_level'])

/usr/local/lib/python3.6/dist-packages/textgenrnn/textgenrnn.py in train_from_largetext_file(self, file_path, new_model, **kwargs) 328 if new_model: 329 self.train_new_model( --> 330 texts, single_text=True, **kwargs) 331 else: 332 self.train_on_texts(texts, single_text=True, **kwargs)

/usr/local/lib/python3.6/dist-packages/textgenrnn/textgenrnn.py in train_new_model(self, texts, context_labels, num_epochs, gen_epochs, batch_size, dropout, validation, save_epochs, multi_gpu, **kwargs) 289 save_epochs=save_epochs, 290 multi_gpu=multi_gpu, --> 291 **kwargs) 292 293 def save(self, weights_path="textgenrnn_weights_saved.hdf5"):

[mrciolino]

+1 - I've had quite a few problems getting things to run but pip installing from the repo helps a lot.

pip install -q git+git://github.com/minimaxir/textgenrnn.git