PicklingError due to recent Caffe update

BVLC/caffe#6219 fixed some pickling bug that occurred due to the namespace caffe_pb2 was in. That fix however causes the following error when loading DIGITS:

Caught PicklingError while saving job 20171212-140813-6b62: Can't pickle <class 'caffe.proto.caffe_pb2.NetParameter'>: it's not the same object as caffe.proto.caffe_pb2.NetParameter

~Consequently, none of the existing models can load successfully.~ Is it possible for DIGITS to handle this exception and fix the jobs on-the-fly during loading?

Correction: I double checked and existing models seem to load and be okay. Any new ones however, fail to save!

To reproduce, create a job using a Caffe backend, shut down the server, update and recompile Caffe, then restart DIGITS and try to clone a job. It will appear in the models list, but will fail to save (there will be no status.pickle).

BTW, this bug has a long story https://github.com/NVIDIA/DIGITS/issues/306 Here is a patch that fixes loading of saved jobs https://github.com/uhfband/DIGITS/commit/59faee35442191356e43f51df87bd9253c1f6f1f It's not perfect, but it works for me.

I have the same problem :( and try import PYTHONPATH but did NOT work

https://github.com/uhfband/DIGITS/commit/59faee35442191356e43f51df87bd9253c1f6f1f#diff-0d1ba4e35115631664436b13944953fd This one worked for me. Replace all the 'import caffe_pb2' with 'import caffe.proto.caffe_pb2 as caffe_pb2'.

@uhfband The replace works for me, thanks a lot!

not work for me!!! did this problem fixed? I installed DIGITS from source

I also encountered this problem. The patch by uhfband seems to solve it properly. Thank you uhfband !