Training failing on Microsoft COCO dataset

[umerebryx]

Hey Guys,

I am trying to train a custom model using a style image (resolution 600x600) and coco dataset. I am running train.py script on gpu. I run into following error:

Setting env variable for CUDA
num traning images: 82783
82783 iterations, 2 epochs
epoch 0
Traceback (most recent call last):
  File "train.py", line 124, in <module>
    x[j] = load_image(imagepaths[i*batchsize + j], image_size)
  File "train.py", line 23, in load_image
    return xp.asarray(image, dtype=np.float32).transpose(2, 0, 1)
  File "/usr/local/lib/python2.7/dist-packages/cupy/creation/from_data.py", line 47, in asarray
    return cupy.array(a, dtype=dtype, copy=False)
  File "/usr/local/lib/python2.7/dist-packages/cupy/creation/from_data.py", line 27, in array
    return core.array(obj, dtype, copy, ndmin)
  File "cupy/core/core.pyx", line 1479, in cupy.core.core.array (cupy/core/core.cpp:49697)
  File "cupy/core/core.pyx", line 1493, in cupy.core.core.array (cupy/core/core.cpp:49313)
  File "/usr/lib/python2.7/dist-packages/PIL/Image.py", line 528, in __getattr__
    raise AttributeError(name)
AttributeError: __float__ 

Any Idea what is causing this? It should be straight forward. Thanks in advance.

[ttoinou]

I had an error once when training because of corrupted jpg files (bad file copy / zip extraction maybe). If that's your case you juste have to wrap the load_image with try: except: It doesn't matter if one image doesn't works as long as your crunching continues

[umerebryx]

Thanks for your input, I tried the try catch routine but it is failing for every image. All these can't be bad. Any other suggestions?

[6o6o]

See here: https://github.com/yusuketomoto/chainer-fast-neuralstyle/issues/40