Error in coco test

[fanq15]

#49 I can train on coco dataset successfully, but when testing, there is always a bug. I have changed the corresponding .yaml file, but it doesn't work.

  dataset: coco
  dataset_path: "./data/coco"
  image_set: valminusminival2014
  root_path: "./data"
  test_image_set: minival2014

The bug is:

[14:21:35] include/dmlc/logging.h:300: [14:21:35] src/io/local_filesys.cc:154: Check failed: allow_null  LocalFileSystem: fail to open "./output/fcis/coco/resnet_v1_101_coco_fcis_end2end_ohem/minival2014/../valminusminival2014/e2e-0008.params"

Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc2io15LocalFileSystem4OpenERKNS0_3URIEPKcb+0x459) [0x7f4512ab8ac9]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc6Stream6CreateEPKcS2_b+0x3a) [0x7f4512ab0f2a]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(MXNDArrayLoad+0x242) [0x7f45126b2142]
[bt] (3) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7f452a8edadc]
[bt] (4) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x1fc) [0x7f452a8ed40c]
[bt] (5) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(_ctypes_callproc+0x48e) [0x7f452ab045fe]
[bt] (6) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(+0x15f9e) [0x7f452ab05f9e]
[bt] (7) python(PyEval_EvalFrameEx+0x98d) [0x5244dd]
[bt] (8) python(PyEval_EvalCodeEx+0x2b1) [0x555551]
[bt] (9) python(PyEval_EvalFrameEx+0x1a10) [0x525560]

Traceback (most recent call last):
  File "experiments/fcis/fcis_end2end_train_test.py", line 14, in <module>
    test.main()
  File "experiments/fcis/../../fcis/test.py", line 52, in main
    args.vis, args.ignore_cache, args.shuffle, config.TEST.HAS_RPN, config.dataset.proposal, args.thresh, logger=logger, output_path=final_output_path)
  File "experiments/fcis/../../fcis/function/test_fcis.py", line 43, in test_fcis
    arg_params, aux_params = load_param(prefix, epoch, process=True)
  File "experiments/fcis/../../fcis/../lib/utils/load_model.py", line 49, in load_param
    arg_params, aux_params = load_checkpoint(prefix, epoch)
  File "experiments/fcis/../../fcis/../lib/utils/load_model.py", line 15, in load_checkpoint
    save_dict = mx.nd.load('%s-%04d.params' % (prefix, epoch))
  File "/usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/ndarray.py", line 1808, in load
    ctypes.byref(names)))
  File "/usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/base.py", line 78, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [14:21:35] src/io/local_filesys.cc:154: Check failed: allow_null  LocalFileSystem: fail to open "./output/fcis/coco/resnet_v1_101_coco_fcis_end2end_ohem/minival2014/../valminusminival2014/e2e-0008.params"

Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc2io15LocalFileSystem4OpenERKNS0_3URIEPKcb+0x459) [0x7f4512ab8ac9]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc6Stream6CreateEPKcS2_b+0x3a) [0x7f4512ab0f2a]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet-0.9.5-py2.7.egg/mxnet/libmxnet.so(MXNDArrayLoad+0x242) [0x7f45126b2142]
[bt] (3) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7f452a8edadc]
[bt] (4) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x1fc) [0x7f452a8ed40c]
[bt] (5) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(_ctypes_callproc+0x48e) [0x7f452ab045fe]
[bt] (6) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(+0x15f9e) [0x7f452ab05f9e]
[bt] (7) python(PyEval_EvalFrameEx+0x98d) [0x5244dd]
[bt] (8) python(PyEval_EvalCodeEx+0x2b1) [0x555551]
[bt] (9) python(PyEval_EvalFrameEx+0x1a10) [0x525560]

[liyi14]

Does there exist a file at ./output/fcis/coco/resnet_v1_101_coco_fcis_end2end_ohem/minival2014/../valminusminival2014/e2e-0008.params?

[fanq15]

No, no matter how I change the training data and testing data, it can't produce it.

[tensorboy]

Did you fix that?

[chrieke]

".../valminusminival2014/e2e-0008.params" is your model you want to test on but it doesn't exist. Look in the output folder which .params file actually exist, maybe you didn't train until the 8th epoch? It creates it's own .params file for every epoch during the training.

You can change the epoch you want to test on in the .yaml file under testepoch.

[liyi14]

Did you change the training epoch while didn't change the test epoch in .yaml file?