attention-lvcsr
attention-lvcsr copied to clipboard
rizar
decoding issue
Hello, I have an issue when I tried to decode the model for TLE.
export THEANO_FLAGS=mode=FAST_RUN,device=gpu3,floatX=float32; $LVSR/exp/wsj/decode_tle.sh wsj_reward6 test 200
Here is the log:
- MODEL=wsj_reward6
- PART=test
- BEAM_SIZE=200
- LM=nolm
- LM_PATH=data/local/nist_lm/wsj_trigram_no_bos
- ls wsj_reward6/reports test_eval92_nolm_200 test_nolm_200
- LM_CONF='monitoring.search.beam_size 200'
- '[' '!' nolm == nolm ']'
- /speech05-ssd/sykim/attention-lvcsr/bin/run.py search --part=test --report wsj_reward6/reports/test_nolm_200 wsj_reward6/annealing1_best_ll.zip /speech05-ssd/sykim/attention-lvcsr/exp/wsj/configs/wsj_reward6.yaml vocabulary data/local/nist_lm/wsj_trigram_no_bos/words.txt monitoring.search.beam_size 200
Using gpu device 3: Tesla K20m (CNMeM is enabled with initial size: 90.0% of memory, cuDNN Version is too old. Update to v5, was 3007.)
2016-06-07 14:49:00,862: pykwalify.core: INFO: validation.valid
2016-06-07 14:49:00,867: pykwalify.core: INFO: validation.valid
2016-06-07 14:49:00,872: pykwalify.core: INFO: validation.valid
2016-06-07 14:49:00,877: pykwalify.core: INFO: validation.valid
2016-06-07 14:49:00,884: main: INFO: Config:
{'cmd_args': {'config_changes': <picklable_itertools.extras.equizip object at 0x7fb144a68050>,
'config_path': '/speech05-ssd/sykim/attention-lvcsr/exp/wsj/configs/wsj_reward6.yaml',
'decode_only': None,
'decoded_save': None,
'func': 'search',
'load_path': 'wsj_reward6/annealing1_best_ll.zip',
'nll_only': False,
'params': None,
'part': 'test',
'report': 'wsj_reward6/reports/test_nolm_200',
'seed': 1,
'validate_config': True},
'data': {'add_bos': 1,
'add_eos': True,
'batch_size': 20,
'dataset_class': <class 'lvsr.datasets.h5py.H5PYAudioDataset'>,
'dataset_filename': 'wsj.h5',
'default_sources': ['recordings', 'labels'],
'name_mapping': {'test': 'test_eval92', 'train': 'train_si284', 'valid': 'test_dev93'},
'normalization': '',
'sources_map': {'labels': 'characters', 'recordings': 'fbank_dd', 'uttids': 'uttids'}},
'initialization': {'/recognizer': {'biases_init': Constant(_constant=0.0),
'rec_weights_init': IsotropicGaussian(_mean=0,_std=0.1),
'weights_init': Uniform(_mean=0.0,_width=0.1)},
'/recognizer/generator/readout/post_merge/mlp': {'biases_init': Constant(_constant=-1.0)}},
'monitoring': {'search': {'beam_size': 200, 'char_discount': 0.0, 'round_to_inf': 4.5, 'stop_on': 'patience'},
'search_every_batches': 0,
'search_every_epochs': 1,
'validate_every_batches': 0,
'validate_every_epochs': 1},
'net': {'attention_type': 'content_and_conv',
'bottom': {'activation': <blocks.bricks.simple.Rectifier object at 0x7fb118424510: name=rectifier>,
'bottom_class': <class 'lvsr.bricks.recognizer.SpeechBottom'>,
'dims': []},
'conv_n': 100,
'criterion': {'min_reward': -5, 'name': 'mse_gain'},
'dec_transition': <class 'blocks.bricks.recurrent.GatedRecurrent'>,
'dim_dec': 250,
'dims_bidir': [250, 250, 250, 250],
'enc_transition': <class 'blocks.bricks.recurrent.GatedRecurrent'>,
'lm': {'normalize_am_weights': False},
'max_decoded_length_scale': 3.0,
'post_merge_activation': <blocks.bricks.simple.Rectifier object at 0x7fb1184336d0: name=rectifier>,
'post_merge_dims': [250],
'prior': {'after': 100,
'before': 100,
'initial_begin': 0,
'initial_end': 80,
'max_speed': 4.4,
'min_speed': 2.4,
'type': 'window_around_median'},
'subsample': [1, 1, 2, 2],
'use_states_for_readout': True},
'parent': '$LVSR/exp/wsj/configs/wsj_reward4.yaml',
'regularization': {'dropout': False, 'max_norm': 1.0},
'stages': {'annealing1': {'net': {'lm': None},
'training': {'epsilon': 1e-10, 'num_epochs': 3, 'restart_from': '_best_ll'}},
'annealing2': None,
'main': {'net': {'lm': None}, 'training': {'num_epochs': 15, 'restart_from': '_best_ll'}},
'pretraining': {'net': {'criterion': {'min_reward': -1},
'lm': None,
'prior': {'initial_begin': 0,
'initial_end': 40,
'max_speed': 2.2,
'min_speed': 1.2,
'type': 'expanding'}},
'training': {'num_epochs': 4}}},
'training': {'decay_rate': 0.95,
'epsilon': 1e-08,
'exploration': 'greedy',
'gradient_threshold': 100.0,
'momentum': 0.0,
'rules': ['momentum', 'adadelta'],
'scale': 0.1},
'vocabulary': 'data/local/nist_lm/wsj_trigram_no_bos/words.txt'}
2016-06-07 14:49:00,996: lvsr.main: INFO: Recognizer initialization started
2016-06-07 14:49:54,017: lvsr.main: INFO: Recognizer is initialized
/speech05-ssd/sykim/attention-lvcsr/lvsr/bricks/recognizer.py:493: UserWarning: theano.function was asked to create a function computing outputs given certain inputs, but the provided input variable at index 1 is not part of the computational graph needed to compute the outputs: groundtruth.
To make this warning into an error, you can pass the parameter on_unused_input='raise' to theano.function. To disable it completely, use on_unused_input='ignore'.
on_unused_input='warn')
Traceback (most recent call last):
File "/speech05-ssd/sykim/attention-lvcsr/bin/run.py", line 154, in
getattr(lvsr.main, args.pop('func'))(config, **args) File "/speech05-ssd/sykim/attention-lvcsr/lvsr/main.py", line 781, in search prediction=raw_groundtruth)[:2] File "/speech05-ssd/sykim/attention-lvcsr/lvsr/bricks/recognizer.py", line 493, in analyze on_unused_input='warn') File "/usr/local/lib/python2.7/dist-packages/theano/compile/function.py", line 322, in function output_keys=output_keys) File "/usr/local/lib/python2.7/dist-packages/theano/compile/pfunc.py", line 480, in pfunc output_keys=output_keys) File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1827, in orig_function output_keys=output_keys).create( File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1479, in init accept_inplace) File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 177, in std_fgraph update_mapping=update_mapping) File "/usr/local/lib/python2.7/dist-packages/theano/gof/fg.py", line 182, in init self.import_r(output, reason="init") File "/usr/local/lib/python2.7/dist-packages/theano/gof/fg.py", line 374, in import_r self.import(variable.owner, reason=reason) File "/usr/local/lib/python2.7/dist-packages/theano/gof/fg.py", line 416, in import variable=r) theano.gof.fg.MissingInputError: An input of the graph, used to compute DimShuffle{0,x}(labels), was not provided and not given a value.Use the Theano flag exception_verbosity='high',for more information on this error.
Backtrace when the variable is created:
File "/speech05-ssd/sykim/attention-lvcsr/bin/run.py", line 154, in
attention-lvcsr/lvsr/main.py", line 716, in search recognizer = create_model(config, data, load_path) File "/speech05-ssd/sykim/attention-lvcsr/lvsr/main.py", line 221, in create_model **net_config) File "/speech05-ssd/sykim/attention-lvcsr/lvsr/bricks/recognizer.py", line 360, in init self.single_labels = tensor.lvector('labels')
Hmm, that's strange.
Can you run a postmortem debugger and check if input_variables contains the labels (this one)?
Somehow, a function analyze is not compiled in your case. It just outputs the cost and visualizes the alignment, it is not needed for decoding, but it's very suspicious...
So, I propose to run a debugger like (assuming that you have ipdb; I'm taking the command from your log)
python -mipdb /speech05-ssd/sykim/attention-lvcsr/bin/run.py search --part=test --report wsj_reward6/reports/test_nolm_200 wsj_reward6/annealing1_best_ll.zip /speech05-ssd/sykim/attention-lvcsr/exp/wsj/configs/wsj_reward6.yaml vocabulary data/local/nist_lm/wsj_trigram_no_bos/words.txt monitoring.search.beam_size 200
You can use ... -mpdb ... if you don't want to install ipdb. If you are not familiar with pdb/ipdb, there are a lot of tutorials in internet how to use it.
You'll be able to go up/down the stack and print/inspect the variables.