tracking_via_colorization icon indicating copy to clipboard operation
tracking_via_colorization copied to clipboard

Published 20 hours ago • verified publisher icon wbaek

Issue with data generator in training

Open mattclifford1 opened this issue 5 years ago • 2 comments

Issue with generator exppecting float32 or int64, I believe images are int8. Full traceback below

Traceback (most recent call last): File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call return fn(*args) File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: TypeError: generator yielded an element that did not match the expected structure. The expected structure was (tf.float32, tf.int64), but the yielded element was [array([[[[217], [217], [217], ..., [ 16], [ 17], [ 17]],

    [[217],
     ................
     ...,
     [ 40],
     [ 42],
     [ 44]]]], dtype=uint8), array([[[[ 9],
     [ 9],
     [ 9],
     ...,
     [15],
    .................
     [13],
     [13],
     ...,
     [ 3],
     [ 3],
     [ 3]]]])].

Traceback (most recent call last):

File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 520, in generator_py_func flattened_values = nest.flatten_up_to(output_types, values)

File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow/python/data/util/nest.py", line 398, in flatten_up_to assert_shallow_structure(shallow_tree, input_tree)

File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow/python/data/util/nest.py", line 301, in assert_shallow_structure "Input has type: %s." % type(input_tree))

TypeError: If shallow structure is a sequence, input must also be a sequence. Input has type: <class 'list'>.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 209, in call ret = func(*args)

File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 525, in generator_py_func "element was %s." % (output_types, values))

TypeError: generator yielded an element that did not match the expected structure. The expected structure was (tf.float32, tf.int64), but the yielded element was [array([[[[217], [217], [217], ..., [ 16], .............................. [ 70], ..., [ 46], [ 46], [ 45]],

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "bin/train_colorizer.py", line 128, in main(parsed_args) File "bin/train_colorizer.py", line 108, in main estimator.train(input_fn=input_functions['train'], steps=1000) File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 367, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1158, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1192, in _train_model_default saving_listeners) File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1484, in _train_with_estimator_spec _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss]) File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 754, in run run_metadata=run_metadata) File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1252, in run run_metadata=run_metadata) File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1353, in run raise six.reraise(*original_exc_info) File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1338, in run return self._sess.run(*args, **kwargs) File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1411, in run run_metadata=run_metadata) File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1169, in run return self._sess.run(*args, **kwargs) File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 950, in run run_metadata_ptr) File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1173, in _run feed_dict_tensor, options, run_metadata) File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run run_metadata) File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: TypeError: generator yielded an element that did not match the expected structure. The expected structure was (tf.float32, tf.int64), but the yielded element was [array([[[[217], [217], [217], ..., [ 16], ........................ ..., [ 40], [ 42], [ 44]]]], dtype=uint8), array([[[[ 9], [ 9], [ 9], ..., [15], .............. ..., [ 3], [ 3], [ 3]]]])]. Traceback (most recent call last):

File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 520, in generator_py_func flattened_values = nest.flatten_up_to(output_types, values)

File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow/python/data/util/nest.py", line 398, in flatten_up_to assert_shallow_structure(shallow_tree, input_tree)

File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow/python/data/util/nest.py", line 301, in assert_shallow_structure "Input has type: %s." % type(input_tree))

TypeError: If shallow structure is a sequence, input must also be a sequence. Input has type: <class 'list'>.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow/python/ops/script_ops.py", line 209, in call ret = func(*args)

File "/home/nulightai/python_envs/AI/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 525, in generator_py_func "element was %s." % (output_types, values))

TypeError: generator yielded an element that did not match the expected structure. The expected structure was (tf.float32, tf.int64), but the yielded element was [array([[[[217], [217], [217], ..., [ 16], ................ [ 46], [ 45]],

MultiProcessRunnerZMQ successfully cleaned-up. MultiProcessRunnerZMQ successfully cleaned-up. MultiProcessRunnerZMQ successfully cleaned-up. MultiProcessRunnerZMQ successfully cleaned-up.

mattclifford1 avatar Sep 12 '19 16:09 mattclifford1

Am using tensorflow-gpu version 1.14 if that is any help

mattclifford1 avatar Sep 13 '19 13:09 mattclifford1

Seems to be a problem with output_types = (tf.float32, tf.int64) being sequence and generator not being a sequence.

If I comment out this assertion in the tensorflow package tensorflow/python/data/util/nest.py

Training seems to be running OK

mattclifford1 avatar Sep 16 '19 12:09 mattclifford1