tensorflow.python.framework.errors_impl.DataLossError: corrupted record at 2535454

[yishen-zhao]

Traceback (most recent call last): File "main.py", line 71, in train(args=args) File "/home/sncdbs/net_disk/zn/bert_lstm_crf/python/bert-ner/bert_lstm_ner.py", line 650, in train tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 471, in train_and_evaluate return executor.run() File "/usr/local/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 610, in run return self.run_local() File "/usr/local/lib/python3.6/site-packages/tensorflow/python/estimator/training.py", line 711, in run_local saving_listeners=saving_listeners) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 354, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1207, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1241, in _train_model_default saving_listeners) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1471, in _train_with_estimator_spec _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss]) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 671, in run run_metadata=run_metadata) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1156, in run run_metadata=run_metadata) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1255, in run raise six.reraise(*original_exc_info) File "/usr/local/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/usr/local/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1240, in run return self._sess.run(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1320, in run run_metadata=run_metadata)) File "/usr/local/lib/python3.6/site-packages/tensorflow/contrib/estimator/python/estimator/early_stopping.py", line 482, in after_run if self._should_stop_fn(): File "/usr/local/lib/python3.6/site-packages/tensorflow/contrib/estimator/python/estimator/early_stopping.py", line 398, in stop_if_no_metric_improvement_fn eval_results = read_eval_metrics(eval_dir) File "/usr/local/lib/python3.6/site-packages/tensorflow/contrib/estimator/python/estimator/early_stopping.py", line 336, in read_eval_metrics for event in _summaries(eval_dir): File "/usr/local/lib/python3.6/site-packages/tensorflow/contrib/estimator/python/estimator/early_stopping.py", line 437, in _summaries for event in summary_iterator.summary_iterator(event_file): File "/usr/local/lib/python3.6/site-packages/tensorflow/python/summary/summary_iterator.py", line 68, in summary_iterator for r in tf_record.tf_record_iterator(path): File "/usr/local/lib/python3.6/site-packages/tensorflow/python/lib/io/tf_record.py", line 174, in tf_record_iterator reader.GetNext() File "/usr/local/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 490, in GetNext return _pywrap_tensorflow_internal.PyRecordReader_GetNext(self) tensorflow.python.framework.errors_impl.DataLossError: corrupted record at 2535454

[kanhuimin]

Have you solved this problem? If it is solved, could you reply me?