Error occurred when finalizing GeneratorDataset iterator

[ljpetkovic]

Hello,

I cannot train my own model when running the train_val.py script on macOS in PyCharm (conda env).

System information: tensorflow 2.5.0 Python 3.9 Keras 2.4.3

45/45 [==============================] - ETA: 0s - loss: 2.0532 - accuracy: 0.5686 - truncated_acc: 0.4852 - truncated_loss: 2.3367Traceback (most recent call last):
  File "/Users/ljudmilapetkovic/PycharmProjects/deep-spell-checkr/deep-spell-checkr/train_val.py", line 109, in <module>
    model.fit_generator(train_loader,
  File "/Users/ljudmilapetkovic/.conda/envs/deep-spell-checkr/lib/python3.9/site-packages/keras/engine/training.py", line 1918, in fit_generator
    return self.fit(
  File "/Users/ljudmilapetkovic/.conda/envs/deep-spell-checkr/lib/python3.9/site-packages/keras/engine/training.py", line 1176, in fit
    self._eval_data_handler = data_adapter.get_data_handler(
  File "/Users/ljudmilapetkovic/.conda/envs/deep-spell-checkr/lib/python3.9/site-packages/keras/engine/data_adapter.py", line 1348, in get_data_handler
    return DataHandler(*args, **kwargs)
  File "/Users/ljudmilapetkovic/.conda/envs/deep-spell-checkr/lib/python3.9/site-packages/keras/engine/data_adapter.py", line 1138, in __init__
    self._adapter = adapter_cls(
  File "/Users/ljudmilapetkovic/.conda/envs/deep-spell-checkr/lib/python3.9/site-packages/keras/engine/data_adapter.py", line 793, in __init__
    peek, x = self._peek_and_restore(x)
  File "/Users/ljudmilapetkovic/.conda/envs/deep-spell-checkr/lib/python3.9/site-packages/keras/engine/data_adapter.py", line 850, in _peek_and_restore
    peek = next(x)
  File "/Users/ljudmilapetkovic/PycharmProjects/deep-spell-checkr/deep-spell-checkr/utils.py", line 176, in datagen
    encoder_input, decoder_input = next(inputs)
  File "/Users/ljudmilapetkovic/PycharmProjects/deep-spell-checkr/deep-spell-checkr/utils.py", line 168, in batch
    data_batch[i] = ctable.encode(token, maxlen)
  File "/Users/ljudmilapetkovic/PycharmProjects/deep-spell-checkr/deep-spell-checkr/utils.py", line 44, in encode
    x[i, self.char2index[c]] = 1.0
KeyError: 'k'
2021-06-15 11:46:16.159552: W tensorflow/core/kernels/data/generator_dataset_op.cc:107] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
	 [[{{node PyFunc}}]]

Process finished with exit code 1

The only thing I modified in the script was this block:

train_books = ['train_Q354_dAlemBERT.txt']
val_books = ['val_PERRROT75.txt']

I have tried different solutions suggested here, but without success.

Do you have any idea why this is happening?

[ljpetkovic]

P.S. I also have the Load settings error:

Cannot load settings from file '/Users/ljudmilapetkovic/Library/Application Support/JetBrains/PyCharmCE2021.1/options/actionSummary.xml': content truncated File content will be recreated

Is it related to the original problem above?

[afotonower]

for me, this error happened in the middle of training when I had no space left on my system.

[ljpetkovic]

I installed anaconda on my 5 TB external hard drive and tried to run the programme from there, but I still receive the same Error occurred when finalizing GeneratorDataset iterator error with the Key Error: '\\':

57/57 [==============================] - ETA: 0s - loss: 1.9530 - accuracy: 0.5800 - truncated_acc: 0.4953 - truncated_loss: 2.23532021-06-20 16:37:19.874595: W tensorflow/core/framework/op_kernel.cc:1755] Unknown: KeyError: '\\'
Traceback (most recent call last):

  File "/Volumes/LaCie/.conda/envs/spell/lib/python3.9/site-packages/tensorflow/python/ops/script_ops.py", line 249, in __call__
    ret = func(*args)

  File "/Volumes/LaCie/.conda/envs/spell/lib/python3.9/site-packages/tensorflow/python/autograph/impl/api.py", line 645, in wrapper
    return func(*args, **kwargs)

  File "/Volumes/LaCie/.conda/envs/spell/lib/python3.9/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 961, in generator_py_func
    values = next(generator_state.get_iterator(iterator_id))

  File "/Volumes/LaCie/.conda/envs/spell/lib/python3.9/site-packages/keras/engine/data_adapter.py", line 821, in wrapped_generator
    for data in generator_fn():

  File "/Volumes/LaCie/PycharmProjects/spell/deep-spell-checkr/utils.py", line 176, in datagen
    encoder_input, decoder_input = next(inputs)

  File "/Volumes/LaCie/PycharmProjects/spell/deep-spell-checkr/utils.py", line 168, in batch
    data_batch[i] = ctable.encode(token, maxlen)

  File "/Volumes/LaCie/PycharmProjects/spell/deep-spell-checkr/utils.py", line 44, in encode
    x[i, self.char2index[c]] = 1.0

KeyError: '\\'


Traceback (most recent call last):
  File "/Volumes/LaCie/PycharmProjects/spell/deep-spell-checkr/train_val.py", line 110, in <module>
    model.fit_generator(train_loader,
  File "/Volumes/LaCie/.conda/envs/spell/lib/python3.9/site-packages/keras/engine/training.py", line 1918, in fit_generator
    return self.fit(
  File "/Volumes/LaCie/.conda/envs/spell/lib/python3.9/site-packages/keras/engine/training.py", line 1189, in fit
    val_logs = self.evaluate(
  File "/Volumes/LaCie/.conda/envs/spell/lib/python3.9/site-packages/keras/engine/training.py", line 1464, in evaluate
    tmp_logs = self.test_function(iterator)
  File "/Volumes/LaCie/.conda/envs/spell/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 889, in __call__
    result = self._call(*args, **kwds)
  File "/Volumes/LaCie/.conda/envs/spell/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py", line 924, in _call
    results = self._stateful_fn(*args, **kwds)
  File "/Volumes/LaCie/.conda/envs/spell/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 3023, in __call__
    return graph_function._call_flat(
  File "/Volumes/LaCie/.conda/envs/spell/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 1960, in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "/Volumes/LaCie/.conda/envs/spell/lib/python3.9/site-packages/tensorflow/python/eager/function.py", line 591, in call
    outputs = execute.execute(
  File "/Volumes/LaCie/.conda/envs/spell/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 59, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.UnknownError:  KeyError: '\\'
Traceback (most recent call last):

  File "/Volumes/LaCie/.conda/envs/spell/lib/python3.9/site-packages/tensorflow/python/ops/script_ops.py", line 249, in __call__
    ret = func(*args)

  File "/Volumes/LaCie/.conda/envs/spell/lib/python3.9/site-packages/tensorflow/python/autograph/impl/api.py", line 645, in wrapper
    return func(*args, **kwargs)

  File "/Volumes/LaCie/.conda/envs/spell/lib/python3.9/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 961, in generator_py_func
    values = next(generator_state.get_iterator(iterator_id))

  File "/Volumes/LaCie/.conda/envs/spell/lib/python3.9/site-packages/keras/engine/data_adapter.py", line 821, in wrapped_generator
    for data in generator_fn():

  File "/Volumes/LaCie/PycharmProjects/spell/deep-spell-checkr/utils.py", line 176, in datagen
    encoder_input, decoder_input = next(inputs)

  File "/Volumes/LaCie/PycharmProjects/spell/deep-spell-checkr/utils.py", line 168, in batch
    data_batch[i] = ctable.encode(token, maxlen)

  File "/Volumes/LaCie/PycharmProjects/spell/deep-spell-checkr/utils.py", line 44, in encode
    x[i, self.char2index[c]] = 1.0

KeyError: '\\'


	 [[{{node PyFunc}}]]
	 [[IteratorGetNext]] [Op:__inference_test_function_10400]

Function call stack:
test_function

2021-06-20 16:37:21.520890: W tensorflow/core/kernels/data/generator_dataset_op.cc:107] Error occurred when finalizing GeneratorDataset iterator: Failed precondition: Python interpreter state is not initialized. The process may be terminated.
	 [[{{node PyFunc}}]]

Process finished with exit code 1