ORQA/REALM - Invalid argument: indices[0] = 10311577 is not in

[ioannist]

I am trying to run Orqa finetuning with Realm models on my own data set. I have created the blocks.tfr file and used --num_block_records=154800 which is the number of rows in my records file.

I launch fine-tuning with

python -m language.orqa.experiments.orqa_experiment --retriever_module_path=$(pwd)/language/orqa/gs/realm-data/cc_news_pretrained/embedder --reader_module_path=$(pwd)/language/orqa/gs/realm-data/cc_news_pretrained/bert --block_records_path=$(pwd)/language/orqa/gs/orqa-data/trump-blocks/blocks.tfr --num_block_records=227246 --data_root=$(pwd)/language/orqa/gs/orqa-data/resplit --model_dir=$(pwd)/language/orqa/models/orqa_trump_model_from_realm --dataset_name=TrumpQuestions --num_train_steps=$(( 3417 * 20 )) --save_checkpoints_steps=1000

Everything seems to go well in the first ten minutes, but once shuffle buffer filling completes, I get an error

2020-10-13 13:06:30.130961: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:199] Shuffle buffer filled.
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/orqa/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1367, in _do_call
    return fn(*args)
  File "/home/ubuntu/anaconda3/envs/orqa/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1352, in _run_fn
    target_list, run_metadata)
  File "/home/ubuntu/anaconda3/envs/orqa/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1445, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: indices[0] = 10311577 is not in [0, 227246)
         [[{{node GatherV2_2}}]]
  (1) Invalid argument: indices[0] = 10311577 is not in [0, 227246)
         [[{{node GatherV2_2}}]]
         [[reader/RaggedRange_4/_8285]]
0 successful operations.
0 derived errors ignored.
....
....
Errors may have originated from an input operation.
Input Source operations connected to node GatherV2_2:
 blocks/read (defined at /ioannis/language/language/orqa/models/orqa_model.py:147)
 Squeeze (defined at /ioannis/language/language/orqa/models/orqa_model.py:129)

Input Source operations connected to node GatherV2_2:
 blocks/read (defined at /ioannis/language/language/orqa/models/orqa_model.py:147)
 Squeeze (defined at /ioannis/language/language/orqa/models/orqa_model.py:129)

[mchari]

@ioannist , were you able to resolve this issue ? how did you create the tfr file ? i am starting to look into this and wondered if there was any quick reference that would help me get going. thanks!

[ioannist]

@michari no, unfortunately not :(

[mchari]

@ioannist , any guidance on how to create a working tfr file ?

[ioannist]

can't recall exactly but I think I created a raw text file in the same format as the Wikipedia raw text file, and then I ran language.preprocessing.preprocess_wiki_extractor

[johnbager]

@ioannist ,hello,i’m sorry to bother you. Could you share me about how do you deal one doc to tfr。（what about language.preprocessing.preprocess_wiki_extractor)

[Kamel773]

@ioannist I have the same issues. Have you solved it?

[Kamel773]

cc @kentonl
I am working on fine tuning ORQA in my own dataset, and I have the same issues. We appreciate your help in solving this issues.

Kind regards,