Issue while training WordClassifier-CTC using IAM Dataset

[yasersakkaf]

trafficstars

---

InvalidArgumentError Traceback (most recent call last) /data/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args) 1360 try: -> 1361 return fn(*args) 1362 except errors.OpError as e:

/data/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata) 1339 return tf_session.TF_Run(session, options, feed_dict, fetch_list, -> 1340 target_list, status, run_metadata) 1341

/data/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py in exit(self, type_arg, value_arg, traceback_arg) 515 compat.as_text(c_api.TF_Message(self.status.status)), --> 516 c_api.TF_GetCode(self.status.status)) 517 # Delete the underlying status object from memory otherwise it stays alive

InvalidArgumentError: Cannot parse tensor from proto: dtype: DT_UINT8 tensor_shape { dim { size: 1 } } tensor_content: "\000\000\000\000"

 [[Node: DMT/_239 = Const[dtype=DT_UINT8, value=<Invalid TensorProto: dtype: DT_UINT8 tensor_shape { dim { size: 1 } } tensor_content: "\000\000\000\000">, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

During handling of the above exception, another exception occurred:

InvalidArgumentError Traceback (most recent call last) in () 8 for i_batch in range(TRAIN_STEPS): 9 fd = train_iterator.next_feed(BATCH_SIZE) ---> 10 train_step.run(fd) 11 12 if i_batch % LOSS_ITER == 0:

/data/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in run(self, feed_dict, session) 2283 none, the default session will be used. 2284 """ -> 2285 _run_using_default_session(self, feed_dict, self.graph, session) 2286 2287 _gradient_registry = registry.Registry("gradient")

/data/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in _run_using_default_session(operation, feed_dict, graph, session) 4934 "the operation's graph is different from the session's " 4935 "graph.") -> 4936 session.run(operation, feed_dict) 4937 4938

/data/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata) 903 try: 904 result = self._run(None, fetches, feed_dict, options_ptr, --> 905 run_metadata_ptr) 906 if run_metadata: 907 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/data/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata) 1135 if final_fetches or final_targets or (handle and feed_dict_tensor): 1136 results = self._do_run(handle, final_targets, final_fetches, -> 1137 feed_dict_tensor, options, run_metadata) 1138 else: 1139 results = []

/data/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata) 1353 if handle is None: 1354 return self._do_call(_run_fn, self._session, feeds, fetches, targets, -> 1355 options, run_metadata) 1356 else: 1357 return self._do_call(_prun_fn, self._session, handle, feeds, fetches)

/data/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args) 1372 except KeyError: 1373 pass -> 1374 raise type(e)(node_def, op, message) 1375 1376 def _extend_graph(self):

InvalidArgumentError: Cannot parse tensor from proto: dtype: DT_UINT8 tensor_shape { dim { size: 1 } } tensor_content: "\000\000\000\000"

 [[Node: DMT/_239 = Const[dtype=DT_UINT8, value=<Invalid TensorProto: dtype: DT_UINT8 tensor_shape { dim { size: 1 } } tensor_content: "\000\000\000\000">, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

I don't quite understand why I am getting this error.

[Breta01]

I not sure what cause this error either. Can you test if this happens for all words or for only some of them?

Also make sure you change char_size variable to have size corresponding to length of IAM characters array.

[yasersakkaf]

I will test it for sure. I didn't change the char_size variable. That maybe the issue. Thanks for responding

[yasersakkaf]

I changed the char_size, but no progress. The error seems to appear for all the images

[yasersakkaf]

I think it is related to tensorflow version. In my local machine I have v1.4.0, it is working fine there. In the GPU that I am using the version is 1.6.0. and I get the error.

[Breta01]

That's true, I also get error on TensorFlow 1.6.0+ (also on non-GPU version). I don't know what is the breaking change. It would be great if you could investigate this further. I'm busy right now and I won't have a chance to get to it in next two weeks.