machine translation example crashes

[thompsonb]

I'm getting the following errors when I try to run the blocks machine translation example (repost from blocks-users, maybe this is a real bug?)

python -m machine_translation < ... > INFO:machine_translation:Total number of parameters: 37 INFO:machine_translation:Building model INFO:machine_translation:Initializing extensions INFO:machine_translation:Building sampling model INFO:machine_translation:Building sampler INFO:machine_translation:Building bleu validator INFO:machine_translation:Initializing training algorithm INFO:blocks.algorithms:Taking the cost gradient INFO:blocks.algorithms:The cost gradient computation graph is built INFO:machine_translation:Initializing main loop INFO:blocks.main_loop:Entered the main loop INFO:blocks.algorithms:Initializing the training algorithm <<!! BUG IN FGRAPH.REPLACE OR A LISTENER !!>> <type 'exceptions.Exception'> Constant{-1} is already owned by another fgraph local_shape_to_shape_i ERROR (theano.gof.opt): Optimization failure due to: local_shape_to_shape_i ERROR:theano.gof.opt:Optimization failure due to: local_shape_to_shape_i ERROR (theano.gof.opt): node: Shape(for{cpu,grad_of_att_trans_do_apply_scan}.20) ERROR:theano.gof.opt:node: Shape(for{cpu,grad_of_att_trans_do_apply_scan}.20) ERROR (theano.gof.opt): TRACEBACK: ERROR:theano.gof.opt:TRACEBACK: ERROR (theano.gof.opt): Traceback (most recent call last): File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 1822, in process_node fgraph.replace_all_validate(repl_pairs, reason=lopt) File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/toolbox.py", line 306, in replace_all_validate fgraph.replace(r, new_r, reason=reason, verbose=False) File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/fg.py", line 592, in replace self.change_input(node, i, new_r, reason=reason) File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/fg.py", line 530, in change_input self.import_r(new_r, reason=reason) File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/fg.py", line 363, in import_r self.import(variable.owner, reason=reason) File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/fg.py", line 394, in import raise Exception("%s is already owned by another fgraph" % r) Exception: Constant{-1} is already owned by another fgraph

ERROR:theano.gof.opt:Traceback (most recent call last): File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 1822, in process_node fgraph.replace_all_validate(repl_pairs, reason=lopt) File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/toolbox.py", line 306, in replace_all_validate fgraph.replace(r, new_r, reason=reason, verbose=False) File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/fg.py", line 592, in replace self.change_input(node, i, new_r, reason=reason) File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/fg.py", line 530, in change_input self.import_r(new_r, reason=reason) File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/fg.py", line 363, in import_r self.import(variable.owner, reason=reason) File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/fg.py", line 394, in import raise Exception("%s is already owned by another fgraph" % r) Exception: Constant{-1} is already owned by another fgraph

ERROR (theano.gof.opt): Optimization failure due to: local_subtensor_lift ERROR:theano.gof.opt:Optimization failure due to: local_subtensor_lift ERROR (theano.gof.opt): node: Subtensor{int64}(MakeVector{dtype='int64'}.0, Constant{2}) ERROR:theano.gof.opt:node: Subtensor{int64}(MakeVector{dtype='int64'}.0, Constant{2}) ERROR (theano.gof.opt): TRACEBACK: ERROR:theano.gof.opt:TRACEBACK: ERROR (theano.gof.opt): Traceback (most recent call last): File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 1787, in process_node replacements = lopt.transform(node) File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/tensor/opt.py", line 2559, in local_subtensor_lift if not u.owner or len(u.clients) > 1: AttributeError: 'TensorVariable' object has no attribute 'clients'

ERROR:theano.gof.opt:Traceback (most recent call last): File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 1787, in process_node replacements = lopt.transform(node) File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/tensor/opt.py", line 2559, in local_subtensor_lift if not u.owner or len(u.clients) > 1: AttributeError: 'TensorVariable' object has no attribute 'clients'

< ... many more AttributeErrors for local_add_canonizer, local_mul_canonizer, etc >

ERROR:blocks.main_loop:Error occured during training.

Blocks will attempt to run on_error extensions, potentially saving data, before exiting and reraising the error. Note that the usual after_training extensions will not be run. The original error will be re-raised and also stored in the training log. Press CTRL + C to halt Blocks immediately. Traceback (most recent call last): File "/home/bjt/anaconda/lib/python2.7/runpy.py", line 162, in _run_module_as_main "main", fname, loader, pkg_name) File "/home/bjt/anaconda/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/state/partition1/localdata/bjt/blocks-examples/machine_translation/main.py", line 41, in get_dev_stream(*_configuration), args.bokeh) File "machine_translation/init.py", line 175, in main main_loop.run() File "/home/bjt/anaconda/lib/python2.7/site-packages/blocks/main_loop.py", line 197, in run reraise_as(e) File "/home/bjt/anaconda/lib/python2.7/site-packages/blocks/utils/init.py", line 225, in reraise_as six.reraise(type(new_exc), new_exc, orig_exc_traceback) File "/home/bjt/anaconda/lib/python2.7/site-packages/blocks/main_loop.py", line 172, in run self.algorithm.initialize() File "/home/bjt/anaconda/lib/python2.7/site-packages/blocks/algorithms/init.py", line 253, in initialize self.inputs, [], updates=all_updates, *_self.theano_func_kwargs) File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/compile/function.py", line 317, in function output_keys=output_keys) File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/compile/pfunc.py", line 526, in pfunc output_keys=output_keys) File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", line 1777, in orig_function output_keys=output_keys).create( File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", line 1457, in init optimizer_profile = optimizer(fgraph) File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 101, in call return self.optimize(fgraph) File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 89, in optimize ret = self.apply(fgraph, _args, *_kwargs) File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 230, in apply sub_prof = optimizer.optimize(fgraph) File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 89, in optimize ret = self.apply(fgraph, _args, *_kwargs) File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 2211, in apply lopt_change = self.process_node(fgraph, node, lopt) File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 1830, in process_node self.failure_callback(e, self, repl_pairs, lopt, node) File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 1688, in warn_inplace return NavigatorOptimizer.warn(exc, nav, repl_pairs, local_opt, node) File "/home/bjt/anaconda/lib/python2.7/site-packages/theano/gof/opt.py", line 1674, in warn raise exc AssertionError:

Original exception: AssertionError

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: Here are the versions of things that seems relevant:

python --version Python 2.7.10 :: Anaconda 1.9.1 (64-bit)

python -c "import theano; print theano.version" 0.7.0.dev-80197cf574b62fa7d869f4de800700d2f75b7f4e

pip install git+git://github.com/mila-udem/blocks.git -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt Collecting git+git://github.com/mila-udem/blocks.git Cloning git://github.com/mila-udem/blocks.git to /tmp/pip-VSApNQ-build Requirement already satisfied (use --upgrade to upgrade): blocks==0.1.1 from git+git://github.com/mila-udem/blocks.git in /home/bjt/anaconda/lib/python2.7/site-packages Requirement already satisfied (use --upgrade to upgrade): numpy==1.9.3 in /home/bjt/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): picklable-itertools==0.1.1 in /home/bjt/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 2)) Requirement already satisfied (use --upgrade to upgrade): progressbar2==2.7.3 in /home/bjt/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 3)) Requirement already satisfied (use --upgrade to upgrade): pyyaml==3.11 in /home/bjt/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 4)) Requirement already satisfied (use --upgrade to upgrade): six==1.9.0 in /home/bjt/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 5)) Requirement already satisfied (use --upgrade to upgrade): toolz==0.7.2 in /home/bjt/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 6)) Requirement already satisfied (use --upgrade to upgrade): theano from git+https://github.com/Theano/Theano.git#egg=theano in /home/bjt/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 9)) Requirement already satisfied (use --upgrade to upgrade): fuel from git+https://github.com/mila-udem/fuel#egg=fuel in /home/bjt/anaconda/lib/python2.7/site-packages (from -r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 12)) Requirement already satisfied (use --upgrade to upgrade): scipy>=0.11 in /home/bjt/anaconda/lib/python2.7/site-packages (from theano->-r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 9)) Requirement already satisfied (use --upgrade to upgrade): h5py in /home/bjt/anaconda/lib/python2.7/site-packages (from fuel->-r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 12)) Requirement already satisfied (use --upgrade to upgrade): tables in /home/bjt/anaconda/lib/python2.7/site-packages (from fuel->-r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 12)) Requirement already satisfied (use --upgrade to upgrade): pyzmq in /home/bjt/anaconda/lib/python2.7/site-packages (from fuel->-r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 12)) Requirement already satisfied (use --upgrade to upgrade): pillow in /home/bjt/anaconda/lib/python2.7/site-packages (from fuel->-r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 12)) Requirement already satisfied (use --upgrade to upgrade): requests in /home/bjt/anaconda/lib/python2.7/site-packages (from fuel->-r https://raw.githubusercontent.com/mila-udem/blocks/master/requirements.txt (line 12)

nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2014 NVIDIA Corporation Built on Wed_Aug_27_10:36:36_CDT_2014 Cuda compilation tools, release 6.5, V6.5.16

[thompsonb]

It looks like a recent Theano change broke things in blocks... I reverted to the Oct 20 version of Theano (9ee4918c0b418ae8863dc0eaab0e220564d6d038) and everything works.

[shuboma]

I also encounter this problem, can you fix it？

[sjtufs]

@thompsonb This solved my problem. Thanks!

[shuboma]

where and how did you install the Oct 20 version of Theano? @sjtufs @thompsonb

[sjtufs]

@shuboma Hi. You just have to clone the Theano repo, and git checkout 9ee4918c0b418ae8863dc0eaab0e220564d6d038 then use python setup.py install to install.

[shuboma]

thanks! it works! @sjtufs @thompsonb

[guxd]

I encountered the same problem! I reverted theano to Oct 20. The first time that the theano was re-installed, it runs OK. but when I re-run the program, it crashes again! I think the problem should be fixed according to the latest theano.

[rizar]

I just ran with the latest Blocks and Theano: it works perfectly! Can you guys please doublecheck?