About Evaluation

[mgunay15]

For the evaluation I run the given command in readme with "--test" parameter but it gives "Index out of range" error. What might be the cause?

Testing on epoch 25... Traceback (most recent call last):2 (0.00+0.24), lr 0.003, l = 2.3097, a = 0.5703, avL = 2.5296, avA = 0.6206, g = -1.0000, emL = 2.4391, emA = 0.6370; gqaExperiment File "main.py", line 850, in main() File "main.py", line 777, in main evalRes = runEvaluation(sess, model, data["main"], dataOps, epoch, evalTest = False, getPreds = True) File "main.py", line 258, in runEvaluation minLoss = prevRes["test"]["minLoss"] if prevRes else float("inf")) File "main.py", line 573, in runEpoch imagesBatch = loadImageBatch(data["images"], batch) File "main.py", line 365, in loadImageBatch imageBatch[i, 0:numObjects] = toFile(imageId)["features"][imageId["idx"], 0:numObjects] File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "/home/ec2-user/conda/envs/tf_gpu/lib/python3.6/site-packages/h5py/_hl/dataset.py", line 553, in getitem selection = sel.select(self.shape, args, dsid=self.id) File "/home/ec2-user/conda/envs/tf_gpu/lib/python3.6/site-packages/h5py/_hl/selections.py", line 94, in select sel[args] File "/home/ec2-user/conda/envs/tf_gpu/lib/python3.6/site-packages/h5py/_hl/selections.py", line 261, in getitem start, count, step, scalar = _handle_simple(self.shape,args) File "/home/ec2-user/conda/envs/tf_gpu/lib/python3.6/site-packages/h5py/_hl/selections.py", line 457, in _handle_simple x,y,z = _translate_int(int(arg), length) File "/home/ec2-user/conda/envs/tf_gpu/lib/python3.6/site-packages/h5py/_hl/selections.py", line 477, in _translate_int raise ValueError("Index (%s) out of range (0-%s)" % (exp, length-1)) ValueError: Index (150458) out of range (0-148854)

[dorarad]

Sorry for that! I'm pretty sure that was a bug in https://github.com/stanfordnlp/mac-network/blob/gqa/merge.py. I resolved it since then but if you ran it before there's some chance it still affects your run. I will be able to look into it more in 2 weeks unfortunately, and in the meantime I would recommend trying rerunning merge.py and run the data with --expName "gqaExperiment2" to make sure it generates clean data with the new code. Also see https://github.com/stanfordnlp/mac-network/issues/20. I hope to get back to you with a more detailed solution soon!

[mgunay15]

Thank you!