pre-trained-keras-example icon indicating copy to clipboard operation
pre-trained-keras-example copied to clipboard

Error while training

Open ghimiremukti opened this issue 6 years ago • 4 comments

with this training script:

project_root="......../simpsons_proj/pre-trained-keras-example" model="resnet50" path_to_data=$project_root/"the-simpsons-characters-dataset/simpsons_dataset" path_to_weight_directory=$project_root/"weight_directory" path_to_tensorboard_logdir=$project_root/"tensorboard_direcoty" max_epochs=20000

python train.py
--pretrained_model=$model
--data-dir=$path_to_data
--weight-directory=$path_to_weight_directory
--tensorboard-directory=$path_to_tensorboard_logdir
--epochs=$max_epochs

the training does not proceed with the following error


2018-07-04 15:43:53.353028: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 67 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 74f2:00:00.0, compute capability: 6.0) train.py:177: UserWarning: Update your fit_generator call to the Keras 2 API: fit_generator(<generator..., use_multiprocessing=True, workers=4, validation_data=<generator..., steps_per_epoch=200, epochs=20000, callbacks=[<keras.ca..., validation_steps=10) pickle_safe=True, /usr/local/lib/python2.7/dist-packages/keras/engine/training.py:2087: UserWarning: Using a generator with use_multiprocessing=True and multiple workers may duplicate your data. Please consider using thekeras.utils.Sequence class. UserWarning('Using a generator with use_multiprocessing=True' Traceback (most recent call last): File "train.py", line 177, in <module> pickle_safe=True, File "/usr/local/lib/python2.7/dist-packages/keras/legacy/interfaces.py", line 91, in wrapper return func(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 2224, in fit_generator class_weight=class_weight) File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1877, in train_on_batch class_weight=class_weight) File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1480, in _standardize_user_data exception_prefix='target') File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 113, in _standardize_input_data 'with shape ' + str(data_shape)) ValueError: Error when checking target: expected dense_2 to have 4 dimensions, but got array with shape (64, 42)

ghimiremukti avatar Jul 04 '18 15:07 ghimiremukti

I have the same problem too.

ykg3mp avatar Oct 02 '18 18:10 ykg3mp

Thanks for @ykg3mp and @ghimiremukti we will try to take a look at this sometime soon.

johndgiese avatar Oct 02 '18 18:10 johndgiese

I think I found the problem, at least for vgg19 The code was missing the flattening part in line 31:

output = Flatten()(model_base.output)

Now I am getting accuracy outputs and the neural net is training.

But for @ghimiremukti's issue, resnet50 has the flattening line in it....I would recommend looking at the generated model.pdf to make sure the model structure does have the flattening in the end (should have output of none,42)

ykg3mp avatar Oct 02 '18 19:10 ykg3mp

@ykg3mp and @ghimiremukti thank you for pointing this out! I have just pushed up a fix. Please try pulling the latest repository version with git pull and re-running the training script. Let me know if it resolves the issue for you and I will close out this ticket.

ReeceStevens avatar Oct 03 '18 15:10 ReeceStevens