nary-grn
nary-grn copied to clipboard
freesunshine0316
Flat indices does not index into param
Hi, I am trying to run your implementation of Peng's model, and I came across with the problem of the wrong input indices, shown as the image below, which made me confused. Do you have any ideas? Thank you in advance!
Hi Is that the full log? We basically implemented a while loop by ourselves. It seems that this is an out-of-boundary problem when indexing an tensor (indexing 26 from a tensor with shape [8, 26 x]).
Oh sorry I only cut the traceback part… below is the full log I get after running it again
CUDA_VISIBLE_DEVICES 2
Loading the configuration from config.json
Configurations:
<namespace_utils.Bunch object at 0x7f2318ab3790>
Log file path: logs/G2S.cross_bin_0.log
Loading train set.
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/1/data_graph_1
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/1/data_graph_2
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/2/data_graph_1
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/2/data_graph_2
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/3/data_graph_1
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/3/data_graph_2
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/4/data_graph_1
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/4/data_graph_2
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/1/data_graph_1
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/1/data_graph_2
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/2/data_graph_1
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/2/data_graph_2
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/3/data_graph_1
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/3/data_graph_2
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/4/data_graph_1
/home/gaoyifei/nary-grn/peng_data/drug_gene_var/4/data_graph_2
Number of training samples: 5313/5313
Number of dev samples: 200/200
Collecting vocabs.
Number of words: 15077
Number of allChars: 137
Number of allEdgelabels: 113
word vocab size 15736
Build DataStream ...
Number of instances in trainDataStream: 5313
Number of instances in devDataStream: 200
Number of batches in trainDataStream: 665
Number of batches in devDataStream: 25
/home/gaoyifei/anaconda3/envs/python27/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py:108: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
2020-04-09 20:52:23.134134: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-04-09 20:52:25.170465: W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at gather_nd_op.cc:50 : Invalid argument: flat indices[35, :] = [3, 25] does not index into param (shape: [8,25,100]).
2020-04-09 20:52:25.170476: W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at gather_nd_op.cc:50 : Invalid argument: flat indices[35, :] = [3, 25] does not index into param (shape: [8,25,100]).
2020-04-09 20:52:25.184736: W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at gather_nd_op.cc:50 : Invalid argument: flat indices[60, :] = [3, 50] does not index into param (shape: [8,50,100]).
2020-04-09 20:52:25.184743: W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at gather_nd_op.cc:50 : Invalid argument: flat indices[60, :] = [3, 50] does not index into param (shape: [8,50,100]).
Return from here, just evaluate
Start the training loop.
Traceback (most recent call last):
File "G2S_trainer.py", line 307, in
Caused by op u'Train/Model/rev_encoder/gated_operations/while/GatherNd', defined at:
File "G2S_trainer.py", line 307, in
InvalidArgumentError (see above for traceback): flat indices[35, :] = [3, 25] does not index into param (shape: [8,25,100]). [[Node: Train/Model/rev_encoder/gated_operations/while/GatherNd = GatherNd[Tindices=DT_INT32, Tparams=DT_FLOAT, _class=["loc:@Train...ul_1/f_acc"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](Train/Model/rev_encoder/gated_operations/while/Identity, Train/Model/rev_encoder/gated_operations/while/stack_1)]]
From your log, the model crashes once being executed. We didn't experience any of such situation before. Did you change any part of the code?
@bhcsayx
I'm pretty busy and you don't provide enough information for what has been down, so all I can help is giving you a brief tour on the code does. Hope this can help you a little bit.
Each graph is organized as a list of nodes (1D) and a list of corresponding neighbors (2D). The node is ordered in a topological way, so that it's guaranteed to have processed all the preceding nodes when processing each node.
The while loop iterates left to right on a batch of instances, it first obtains a vertical slice of nodes and their corresponding neighbors, before updating the hidden states for the nodes.
I'm curious how this out-of-boundary indexing happens, which always exceeds by 1 index (e.g. visiting index 25 for a sequence of 25 elements, or visiting index 34 for a sequence of 34 elements). I highly suspect that you made some changes for the datastream part.
@freesunshine0316
Hi, thank you for sharing the repo. I've also encountered the same error for flat indices and I didn't change any of the datastream part. Here's the traceback:
Start the training loop.
0 2021-02-17 13:16:26.152175: W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at gather_nd_op.cc:50 : Invalid argument: flat indices[37, :] = [5, 25] does not index into param (shape: [8,25,100]).
2021-02-17 13:16:26.152176: W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at gather_nd_op.cc:50 : Invalid argument: flat indices[37, :] = [5, 25] does not index into param (shape: [8,25,100]).
2021-02-17 13:16:26.183353: W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at gather_nd_op.cc:50 : Invalid argument: flat indices[50, :] = [5, 73] does not index into param (shape: [8,73,100]).
2021-02-17 13:16:26.183357: W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at gather_nd_op.cc:50 : Invalid argument: flat indices[50, :] = [5, 73] does not index into param (shape: [8,73,100]).
Traceback (most recent call last):
File "G2S_trainer.py", line 307, in <module>
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "G2S_trainer.py", line 232, in main
_, loss_value, _ = train_graph.execute(sess, cur_batch, cur_batch_rev, FLAGS, is_train=True)
File "/home/julia/Desktop/nary-grn/bidir_dag_lstm/G2S_model_graph.py", line 193, in execute
return sess.run([self.accu, self.loss, self.train_op], feed_dict)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 877, in run
run_metadata_ptr)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1100, in _run
feed_dict_tensor, options, run_metadata)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1272, in _do_run
run_metadata)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1291, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: flat indices[37, :] = [5, 25] does not index into param (shape: [8,25,100]).
[[Node: Train/Model/rev_encoder/gated_operations/while/GatherNd_1 = GatherNd[Tindices=DT_INT32, Tparams=DT_FLOAT, _class=["loc:@Train...ul_1/f_acc"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](Train/Model/rev_encoder/gated_operations/while/Switch_1:1, Train/Model/rev_encoder/gated_operations/while/stack_1)]]
Caused by op u'Train/Model/rev_encoder/gated_operations/while/GatherNd_1', defined at:
File "G2S_trainer.py", line 307, in <module>
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "G2S_trainer.py", line 186, in main
char_vocab=char_vocab, options=FLAGS, mode='train')
File "/home/julia/Desktop/nary-grn/bidir_dag_lstm/G2S_model_graph.py", line 62, in __init__
is_training = is_training, options = options)
File "/home/julia/Desktop/nary-grn/bidir_dag_lstm/dag_encoder_utils.py", line 276, in __init__
idx_var.get_shape(),])
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3232, in while_loop
return_same_structure)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2952, in BuildLoop
pred, body, original_loop_vars, loop_vars, shape_invariants)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2887, in _BuildLoop
body_result = body(*packed_vars_for_body)
File "/home/julia/Desktop/nary-grn/bidir_dag_lstm/dag_encoder_utils.py", line 229, in _recurrence
prev_cell = collect_neighbor_node_representations_2D(passage_node_cell, prev_idx)
File "/home/julia/Desktop/nary-grn/bidir_dag_lstm/dag_encoder_utils.py", line 14, in collect_neighbor_node_representations_2D
return tf.gather_nd(representation, indices)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 3052, in gather_nd
"GatherNd", params=params, indices=indices, name=name)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 454, in new_func
return func(*args, **kwargs)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3155, in create_op
op_def=op_def)
File "/home/julia/anaconda3/envs/nary_py2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1717, in __init__
self._traceback = tf_stack.extract_stack()
InvalidArgumentError (see above for traceback): flat indices[37, :] = [5, 25] does not index into param (shape: [8,25,100]).
[[Node: Train/Model/rev_encoder/gated_operations/while/GatherNd_1 = GatherNd[Tindices=DT_INT32, Tparams=DT_FLOAT, _class=["loc:@Train...ul_1/f_acc"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](Train/Model/rev_encoder/gated_operations/while/Switch_1:1, Train/Model/rev_encoder/gated_operations/while/stack_1)]]```
Any suggestions for where to look into? Thank you in advance!
The error(s) are because of the mismatching in the config.json file : For the default configurations ["word_format": "lemma" and "class_num": 2,], the gs_lstm model works fine.
