OutOfRangeError: End of sequence

[missximon]

Hi, I use tiny-imagenet-200 dataset yo train resnet model, but have occurred the problem:

Traceback (most recent call last): File "./resnet_ctl_imagenet_main.py", line 268, in app.run(main) File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 303, in run _run_main(main, args) File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 251, in _run_main sys.exit(main(argv)) File "./resnet_ctl_imagenet_main.py", line 261, in main stats = run(flags.FLAGS) File "./resnet_ctl_imagenet_main.py", line 243, in run resnet_controller.train(evaluate=not flags_obj.skip_eval) File "/home/siwei.zm/mlperf/training/image_classification/tensorflow2/tf2_common/training/controller.py", line 258, in train train_outputs = self.train_fn(steps_per_loop) File "/home/siwei.zm/mlperf/training/image_classification/tensorflow2/tf2_common/training/standard_runnable.py", line 70, in train self.train_loop_fn(self.train_iter, num_steps) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 828, in call result = self._call(*args, **kwds) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 888, in _call return self._stateless_fn(*args, **kwds) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 2943, in call filtered_flat_args, captured_inputs=graph_function.captured_inputs) # pylint: disable=protected-access File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1919, in _call_flat ctx, args, cancellation_manager=cancellation_manager)) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 560, in call ctx=ctx) File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute inputs, attrs, num_outputs) tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence [[{{node while/body/_1/while/IteratorGetNext}}]] [Op:__inference_loop_fn_24053] Function call stack: loop_fn

And I run the code with the script :

python3 ./resnet_ctl_imagenet_main.py
--base_learning_rate=8.5 ' --batch_size=1
--clean
--data_dir=/home/siwei.zm/mlperf/dataset/miniImageNet/tiny-imagenet-200/train
--datasets_num_private_threads=1
--dtype=fp16
--device_warmup_steps=1
--noenable_device_warmup
--enable_eager
--noenable_xla
--epochs_between_evals=1
--noeval_dataset_cache
--eval_offset_epochs=1
--eval_prefetch_batchs=1
--label_smoothing=0.1
--lars_epsilon=0
--log_steps=1
--lr_schedule=polynomial
--model_dir=/home/siwei.zm/mlperf/model/
--momentum=0.9
--num_accumulation_steps=1
--num_classes=200
--num_gpus=1
--optimizer=LARS
--noreport_accuracy_metrics
--single_l2_loss_op
--noskip_eval
--steps_per_loop=100000
--target_accuracy=0.759
--notf_data_experimental_slack
--tf_gpu_thread_mode=gpu_private
--notrace_warmup
--train_epochs=1
--notraining_dataset_cache
--training_prefetch_batchs=1
--nouse_synthetic_data
--warmup_epochs=1
--weight_decay=0.0002

My host is 8-T4-16GB.

I am not good at it. So where is the problem?Can anyone help me? Thanks very much!

[johntran-nv]

@sgpyc can you advise?

[johntran-nv]

The "OutOfRangeError" suggests that maybe something in the code is still assuming >200 classes. Is there a reason why you can't use the full dataset that we use for MLPerf runs? I'm not sure we have engineering bandwidth to support out-of-scope use cases like this.

[hiwotadese]

Closing because the benchmark has retired.