Number of allEdgelabels: 113
word vocab size 15736
Build DataStream ...
Number of instances in trainDataStream: 5313
Number of instances in devDataStream: 200
Number of batches in trainDataStream: 665
Number of batches in devDataStream: 25
Traceback (most recent call last):
File "G2S_trainer.py", line 308, in
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "/root/anaconda3/envs/tf27/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "G2S_trainer.py", line 187, in main
char_vocab=char_vocab, options=FLAGS, mode='train')
File "/home/yaoshengnan/nary-grn-master/bidir_dag_lstm/G2S_model_graph.py", line 55, in init
is_training = is_training, options = options)
File "/home/yaoshengnan/nary-grn-master/bidir_dag_lstm/dag_encoder_utils.py", line 160, in init
w_trans = tf.get_variable("w_trans", [input_dim + edge_dim, options.dag_hidden_dim], dtype=tf.float32)
AttributeError: 'Bunch' object has no attribute 'dag_hidden_dim'
Could you please help on this issue?
Mar 06
'20 05:03
ysn7
Hi,
I updated this repository that introduced this error. I just fix this and you can check out the updates.
Alternatively, you first define "dag_hidden_dim = options.neighbor_vector_dim" first, then change all "options.dag_hidden_dim" into "dag_hidden_dim".
Saving weights, ACCU 0.72 (prev_best) < 0.735 (cur)
Duration 6.431 sec
4700 4800 4900 5000 5100 5200 5300
Step 5319: loss = 0.51 (481.905 sec)
Validation Data Eval:
Dev loss = 13.3517
Dev accu = 0.7300 146/200
Duration 5.849 sec
5400 5500 5600 5700 5800 5900
Step 5984: loss = 0.51 (481.534 sec)
Validation Data Eval:
Dev loss = 12.7348
Dev accu = 0.7650 153/200
Saving weights, ACCU 0.735 (prev_best) < 0.765 (cur)
Duration 6.653 sec
6000 6100 6200 6300 6400 6500 6600
Step 6649: loss = 0.51 (474.455 sec)
Validation Data Eval:
Dev loss = 12.6276
Dev accu = 0.7600 152/200
Duration 5.454 sec
6700 6800 6900 7000 7100 7200 7300 2020-03-08 19:21:38.507817: E tensorflow/core/kernels/check_numerics_op.cc:185] abnormal_detected_host @0x7f2a14a12d00 = {1, 0} Found Inf or NaN global norm.
Traceback (most recent call last):
File "G2S_trainer.py", line 308, in
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "/root/anaconda3/envs/tf27/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "G2S_trainer.py", line 233, in main
_, loss_value, _ = train_graph.execute(sess, cur_batch, cur_batch_rev, FLAGS, is_train=True)
File "/home/yaoshengnan/nary-grn-master/bidir_dag_lstm/G2S_model_graph.py", line 193, in execute
return sess.run([self.accu, self.loss, self.train_op], feed_dict)
File "/root/anaconda3/envs/tf27/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/root/anaconda3/envs/tf27/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/root/anaconda3/envs/tf27/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/root/anaconda3/envs/tf27/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Found Inf or NaN global norm. : Tensor had NaN values
[[node Train/Model/VerifyFinite/CheckNumerics (defined at /home/yaoshengnan/nary-grn-master/bidir_dag_lstm/G2S_model_graph.py:148) ]]
Caused by op u'Train/Model/VerifyFinite/CheckNumerics', defined at:
File "G2S_trainer.py", line 308, in
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "/root/anaconda3/envs/tf27/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "G2S_trainer.py", line 187, in main
char_vocab=char_vocab, options=FLAGS, mode='train')
File "/home/yaoshengnan/nary-grn-master/bidir_dag_lstm/G2S_model_graph.py", line 148, in init
grads, _ = tf.clip_by_global_norm(tf.gradients(self.loss, tvars), clipper)
File "/root/anaconda3/envs/tf27/lib/python2.7/site-packages/tensorflow/python/ops/clip_ops.py", line 271, in clip_by_global_norm
"Found Inf or NaN global norm.")
File "/root/anaconda3/envs/tf27/lib/python2.7/site-packages/tensorflow/python/ops/numerics.py", line 44, in verify_tensor_all_finite
return verify_tensor_all_finite_v2(t, msg, name)
File "/root/anaconda3/envs/tf27/lib/python2.7/site-packages/tensorflow/python/ops/numerics.py", line 62, in verify_tensor_all_finite_v2
verify_input = array_ops.check_numerics(x, message=message)
File "/root/anaconda3/envs/tf27/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 919, in check_numerics
"CheckNumerics", tensor=tensor, message=message, name=name)
File "/root/anaconda3/envs/tf27/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/root/anaconda3/envs/tf27/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/root/anaconda3/envs/tf27/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op
op_def=op_def)
File "/root/anaconda3/envs/tf27/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1801, in init
self._traceback = tf_stack.extract_stack()
InvalidArgumentError (see above for traceback): Found Inf or NaN global norm. : Tensor had NaN values
[[node Train/Model/VerifyFinite/CheckNumerics (defined at /home/yaoshengnan/nary-grn-master/bidir_dag_lstm/G2S_model_graph.py:148) ]]
Hi,it has some new problems.could you please help on this issue? thank you.
Mar 09
'20 06:03
ysn7
I also observed some errors when the model has reached a high accuracy, and I didn't pinpoint how this happened. It might be due to the "op.while" loop, as the GRN model never reported any errors.
ok,thank you!
and when i want to excute "sh decode.sh".it can't find this file.
Loading configurations from logs/G2S.cross_bin_0.config.json
Traceback (most recent call last):
File "G2S_evaluater.py", line 43, in
FLAGS = namespace_utils.load_namespace(model_prefix + ".config.json")
File "/home/yaoshengnan/new_gs/gs_lstm/namespace_utils.py", line 13, in load_namespace
with open(in_path, 'r') as fp:
IOError: [Errno 2] No such file or directory: 'logs/G2S.cross_bin_0.config.json'
1
Mar 15
'20 09:03
ysn7
You can check first which files are generated in the "logs" folder. This is under 5-fold cross validation setting, so 5 models have to be trained with different data portions.