CNTK
CNTK copied to clipboard
microsoft
Failed to convert pre-trained caffe model by run_convert.py.
Following the instructions in "./bindings/python/cntk/contrib/crosstalkcaffe/examples", the "run_convert.py" failed to load caffe model. Tried with VGG16 and NIN, the error messages are similar:
$ python run_convert.py -n NIN -l log_nin.txt
Traceback (most recent call last):
File "run_convert.py", line 42, in
Any suggestions? Thanks!
In the page of CaffeConverter - "https://github.com/Microsoft/CNTK/tree/master/bindings/python/cntk/contrib/crosstalkcaffe", is the third dependency ("protoc" and "caffe_pb2.py") necessary if only running the samples? It seems the error is coming from parsing the ".prototxt" files in the examples.
Hi @SGShen. Yes, you are right, the tool needs the runtime caffe or compiled prototxt file, as the tool README.md suggestion. It's obvious that we can't analysis the network without corresponding definitions. It seems we need to add extra comment in Examples to explain that. Thanks for your feedback. If you have any other questions about this tool, please feel free to discuss here.
Hi, @yuxiaoguo ,
Thank you for your feedback! I will give it a try.
What confused me is, there is already a "caffe_pb2.py" under "adapter/bvlccaffe". When using that to load ".prototxt" in one of the example folders (e.g. "VGG"), it seems to work. The python I wrote is :
import google.protobuf from cntk.contrib.crosstalkcaffe.adapter.bvlccaffe import caffe_pb2 net = caffe_pb2.NetParameter() f = open("./VGG_ILSVRC_19_layers_deploy.prototxt", "r") google.protobuf.text_format.Merge(f.read(), net) f.close() len(net.layer) print (net.layer[1])
It gives the output like:
_name: "conv1_1" type: "Convolution" bottom: "data" top: "conv1_1" convolution_param { num_output: 64 pad: 1 kernel_size: 3 }
But when using "run_contert.py", it seems to fail when parsing any ".prototxt" in the example directory.
When you said 'fail', you mean the program is crash without any further exceptions or with some error message? There are actual two parsing methods, via pb2 or runtime Caffe. If you choose to use pb2, the validation step will be ignored and the convert speed is really slow (may taking minutes...). Thus, maybe you could try runtime Caffe support Dependiency. If you can only use pb2 converting currently, please attach the failure log and system environment. We will help you to address the problem.
Hi, @yuxiaoguo,
The failure occurs when running the example code "run_converter.py" (https://github.com/Microsoft/CNTK/tree/master/bindings/python/cntk/contrib/crosstalkcaffe/examples). The key function is CaffeConverter.from_model() -- the parameter is file path to a global json file; do not see settings for different parsing method. The complete error messages are listed in the first post of this thread.
When digging into the "caffe_pb2.py", i noticed this module parses ".prototxt" just fine. However, not sure why CaffeConverter.from_model() does not work.
Hello,
I followed the instructions for the dependency -- "https://github.com/Microsoft/CNTK/tree/master/bindings/python/cntk/contrib/crosstalkcaffe", built and copied "caffe_pb2.py" to the "adapter/bvlccaffe" directory. That did not make any differences. The error message is the same:
$ python run_convert.py -n NIN
Traceback (most recent call last):
File "run_convert.py", line 42, in
The build of Caffe was successful. What else could be the cause? Thanks!
OK, thanks for your feedback. I will look into it. Would you mind to share your system envs? Python version, Windows/Linux and CNTK version...
@yuxiaoguo , I am running CNTK on Ubuntu 16.04. Python is cntk-py36. CNTK code is pulled from git. Thanks!
This is still a problem, since we cant access the pretrainednetworks .
(cntk-py36) C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py36\Lib\site-packages\cntk\contrib\crosstalkcaffe\examples>python run_convert.py -n ResNet152 Warning: Un-supported bvlccaffe type: scale_conv1-Scale Warning: Un-supported bvlccaffe type: scale2a_branch1-Scale Warning: Un-supported bvlccaffe type: scale2a_branch2a-Scale Warning: Un-supported bvlccaffe type: scale2a_branch2b-Scale Warning: Un-supported bvlccaffe type: scale2a_branch2c-Scale Traceback (most recent call last): File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py36\lib\site-packages\cntk\contrib\crosstalkcaffe\adapter\bvlccaffe\caffeadapter.py", line 531, in _get_layer_type cntk_layer_type = caffeimpl.CAFFE_LAYER_WRAPPER[caffe_layer_type] KeyError: 'Eltwise'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "run_convert.py", line 42, in
(cntk-py36) C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py36\Lib\site-packages\cntk\contrib\crosstalkcaffe\examples>
The output I posted above was only one type of error I got trying to convert. There is another error type that I could fix, where it stoped on attribute "data". that turned out to be on the definition of input layer. I maanged to convert 5 models by replacing the input layer like this: if input is defined like this ( or variations):
input: "data" input_dim: 1 input_dim: 3 input_dim: 224 input_dim: 224
or like this:
layer { name: "input" type: "data" input_param { shape { dim: 1 dim: 3 dim: 224 dim: 224 } } }
then replace it with: layer { name: "input" type: "Input" top: "data" input_param { shape { dim: 1 dim: 3 dim: 224 dim: 224 } } }