pre-trained-keras-example
pre-trained-keras-example copied to clipboard
Error while training
with this training script:
project_root="......../simpsons_proj/pre-trained-keras-example" model="resnet50" path_to_data=$project_root/"the-simpsons-characters-dataset/simpsons_dataset" path_to_weight_directory=$project_root/"weight_directory" path_to_tensorboard_logdir=$project_root/"tensorboard_direcoty" max_epochs=20000
python train.py
--pretrained_model=$model
--data-dir=$path_to_data
--weight-directory=$path_to_weight_directory
--tensorboard-directory=$path_to_tensorboard_logdir
--epochs=$max_epochs
the training does not proceed with the following error
2018-07-04 15:43:53.353028: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 67 MB memory) -> physical GPU (device: 1, name: Tesla P100-PCIE-16GB, pci bus id: 74f2:00:00.0, compute capability: 6.0)
train.py:177: UserWarning: Update your fit_generator
call to the Keras 2 API: fit_generator(<generator..., use_multiprocessing=True, workers=4, validation_data=<generator..., steps_per_epoch=200, epochs=20000, callbacks=[<keras.ca..., validation_steps=10)
pickle_safe=True,
/usr/local/lib/python2.7/dist-packages/keras/engine/training.py:2087: UserWarning: Using a generator with use_multiprocessing=True
and multiple workers may duplicate your data. Please consider using thekeras.utils.Sequence class. UserWarning('Using a generator with
use_multiprocessing=True' Traceback (most recent call last): File "train.py", line 177, in <module> pickle_safe=True, File "/usr/local/lib/python2.7/dist-packages/keras/legacy/interfaces.py", line 91, in wrapper return func(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 2224, in fit_generator class_weight=class_weight) File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1877, in train_on_batch class_weight=class_weight) File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1480, in _standardize_user_data exception_prefix='target') File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 113, in _standardize_input_data 'with shape ' + str(data_shape)) ValueError: Error when checking target: expected dense_2 to have 4 dimensions, but got array with shape (64, 42)
I have the same problem too.
Thanks for @ykg3mp and @ghimiremukti we will try to take a look at this sometime soon.
I think I found the problem, at least for vgg19 The code was missing the flattening part in line 31:
output = Flatten()(model_base.output)
Now I am getting accuracy outputs and the neural net is training.
But for @ghimiremukti's issue, resnet50 has the flattening line in it....I would recommend looking at the generated model.pdf to make sure the model structure does have the flattening in the end (should have output of none,42)
@ykg3mp and @ghimiremukti thank you for pointing this out! I have just pushed up a fix. Please try pulling the latest repository version with git pull
and re-running the training script. Let me know if it resolves the issue for you and I will close out this ticket.