failed prediction only in some samples

[ramtinz]

I'm using a docker container with TF1.15 to run DeepMicrobes. I know it might not have been tested using TF1.15 but that is the compatible one with my platform. It has no problem to do the prediction for many tfrec files (converted from FASTQ files) but has problems with some. Here is the error I get for example:

Traceback (most recent call last):
  File "/root/DeepMicrobes.py", line 382, in <module>
    absl_app.run(main)
  File "/usr/local/lib/python3.8/dist-packages/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.8/dist-packages/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "/root/DeepMicrobes.py", line 362, in main
    classes, probs = paired_report(predict_out,
  File "/root/models/format_prediction.py", line 88, in paired_report
    batch_prob = next(prediction_generator)['probabilities']
  File "/usr/local/lib/python3.8/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 640, in predict
    preds_evaluated = mon_sess.run(predictions)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/training/monitored_session.py", line 750, in run
    return self._sess.run(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1255, in run
    return self._sess.run(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1360, in run
    raise six.reraise(*original_exc_info)
  File "/usr/local/lib/python3.8/dist-packages/six.py", line 719, in reraise
    raise value
  File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1345, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1413, in run
    outputs = _WrappedSession.run(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1176, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/client/session.py", line 955, in run
    result = self._run(None, fetches, feed_dict, options_ptr,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/client/session.py", line 1179, in _run
    results = self._do_run(handle, final_targets, final_fetches,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/client/session.py", line 1358, in _do_run
    return self._do_call(_run_fn, feeds, fetches, targets, options,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.DataLossError: 2 root error(s) found.
  (0) Data loss: corrupted record at 1626132269
         [[node IteratorGetNext (defined at usr/local/lib/python3.8/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]
  (1) Data loss: corrupted record at 1626132269
         [[node IteratorGetNext (defined at usr/local/lib/python3.8/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]]
         [[token_lstm/bidirectional_rnn/bw/bw/Assert/Assert/data_0/_127]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'IteratorGetNext':
  File "root/DeepMicrobes.py", line 382, in <module>
    absl_app.run(main)
  File "usr/local/lib/python3.8/dist-packages/absl/app.py", line 312, in run
    _run_main(main, args)
  File "usr/local/lib/python3.8/dist-packages/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "root/DeepMicrobes.py", line 362, in main
    classes, probs = paired_report(predict_out,
  File "root/models/format_prediction.py", line 88, in paired_report
    batch_prob = next(prediction_generator)['probabilities']
  File "usr/local/lib/python3.8/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 619, in predict
    features, input_hooks = self._get_features_from_input_fn(
  File "usr/local/lib/python3.8/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 996, in _get_features_from_input_fn
    result = self._call_input_fn(input_fn, mode)
  File "usr/local/lib/python3.8/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1116, in _call_input_fn
    return input_fn(**kwargs)
  File "root/DeepMicrobes.py", line 306, in input_fn_predict
    input_fn = input_function_predict_kmer(
  File "root/models/input_pipeline.py", line 147, in input_function_predict_kmer
    batch_features = iterator.get_next()
  File "usr/local/lib/python3.8/dist-packages/tensorflow_core/python/data/ops/iterator_ops.py", line 425, in get_next
    flat_ret = gen_dataset_ops.iterator_get_next(
  File "usr/local/lib/python3.8/dist-packages/tensorflow_core/python/ops/gen_dataset_ops.py", line 2516, in iterator_get_next
    _, _, _op = _op_def_lib._apply_op_helper(
  File "usr/local/lib/python3.8/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 792, in _apply_op_helper
    op = g.create_op(op_type_name, inputs, dtypes=None, name=scope,
  File "usr/local/lib/python3.8/dist-packages/tensorflow_core/python/util/deprecation.py", line 513, in new_func
    return func(*args, **kwargs)
  File "usr/local/lib/python3.8/dist-packages/tensorflow_core/python/framework/ops.py", line 3356, in create_op
    return self._create_op_internal(op_type, inputs, dtypes, input_types, name,
  File "usr/local/lib/python3.8/dist-packages/tensorflow_core/python/framework/ops.py", line 3418, in _create_op_internal
    ret = Operation(
  File "usr/local/lib/python3.8/dist-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()

another error is this one:

Traceback (most recent call last):
  File "/root/DeepMicrobes.py", line 373, in <module>
    absl_app.run(main)
  File "/usr/local/lib/python3.8/dist-packages/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.8/dist-packages/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "/root/DeepMicrobes.py", line 353, in main
    classes, probs = paired_report(predict_out,
  File "/root/models/format_prediction.py", line 89, in paired_report
    batch_prob = average_paired_end(batch_prob, num_classes)
  File "/root/models/format_prediction.py", line 14, in average_paired_end
    prob_matrix = np.mean(np.reshape(prob_matrix, (-1, 4, num_classes)), axis=1)
  File "<__array_function__ internals>", line 5, in reshape
  File "/usr/local/lib/python3.8/dist-packages/numpy/core/fromnumeric.py", line 301, in reshape
    return _wrapfunc(a, 'reshape', newshape, order=order)
  File "/usr/local/lib/python3.8/dist-packages/numpy/core/fromnumeric.py", line 61, in _wrapfunc
    return bound(*args, **kwds)
ValueError: cannot reshape array of size 20072565 into shape (4,2505)

Any idea how to solve them?

[MicrobeLab]

Hi,

The first error seems to originate from data loading. It should be related to some basic process by tensorflow instead of the codes of DeepMicrobes. I'm not sure how to solve it. It may be helpful to check whether the error can be reproduced for the same file.

The second error may be due to the fact that the number of sequences in your tfrec was not a multiple of 4. Since we have the two strands of a pair of reads, the number of sequences should always be a multiple of 4.

[ramtinz]

Hi,

Thank you for the tips. Just to mention that the errors were due to a few incomplete files.