ValueError: ('object too deep for desired array', 'Container name "mean_square_grad_b0"')

[andyyuan78]

ubgpu@ubgpu:~/github/MADE$ sudo python -u trainMADE.py --name mnist_from_paper binarized_mnist 0.01 0 -1 32 Full 300 100 30 False 0 adagrad 0 [8000,8000] 1234 False Output False hinge Orthogonal 0 Using gpu device 0: GeForce GTX 970

Resuming experiment (mnist_from_paper).

Loading dataset [binarized_mnist] ... (Dim:784 Train:50000 Valid:10000 Test:10000) DONE in 0.3176 seconds.

Initializing MADE ... DONE in 947.4197 seconds.

Traceback (most recent call last): File "trainMADE.py", line 293, in load_model_params(model, save_path_experiment) File "trainMADE.py", line 230, in load_model_params model.update_rule.parameters[i].set_value(param.get_value()) File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/var.py", line 132, in set_value self.container.value = value # this will copy a numpy ndarray File "/usr/local/lib/python2.7/dist-packages/theano/gof/link.py", line 343, in set **kwargs) File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/type.py", line 94, in filter_inplace strict, old_data) ValueError: ('object too deep for desired array', 'Container name "mean_square_grad_b0"') ubgpu@ubgpu:~/github/MADE$

[mgermain]

@andyyuan78 thanks for the report.

Looks like a file in the experiment you are trying to resume may be corrupted.

Have you tried to remove the ~/github/MADE/experiments/mnist_from_paper folder and try again? You will lose the progress you've made on the training, but it should fix the problem. If you have any more information please let me know.