Cannot create a tensor proto whose content is larger than 2GB

[hammad001]

For rationale mode, the repository provides code for calculating Question and correct answer pairs BERT embeddings only for train and validation set. While for test set, embeddings are calculated for all question/answer pairs. If we try to extend this functionality to train set, following error occurs:

Traceback (most recent call last):
  File "extract_features.py", line 245, in <module>
    for result in tqdm(estimator.predict(input_fn, yield_single_examples=True)):
  File "/usr/local/lib/python3.5/dist-packages/tqdm/_tqdm.py", line 1022, in __iter__
    for obj in iterable:
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 2437, in predict
    rendezvous.raise_errors()
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/tpu/python/tpu/error_handling.py", line 128, in raise_errors
    six.reraise(typ, value, traceback)
  File "/usr/local/lib/python3.5/dist-packages/six.py", line 693, in reraise
    raise value
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 2431, in predict
    yield_single_examples=yield_single_examples):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 549, in predict
    input_fn, model_fn_lib.ModeKeys.PREDICT)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/estimator/estimator.py", line 1024, in _get_features_from_input_fn
    result = self._call_input_fn(input_fn, mode)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/contrib/tpu/python/tpu/tpu_estimator.py", line 2354, in _call_input_fn
    return input_fn(**kwargs)
  File "/vol/vcr/r2c/data/get_bert_embeddings/vcr_loader.py", line 57, in input_fn
    dtype=tf.int32),
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py", line 207, in constant
    value, dtype=dtype, shape=shape, verify_shape=verify_shape))
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/tensor_util.py", line 506, in make_tensor_proto
    "Cannot create a tensor proto whose content is larger than 2GB.")

[harukaza]

Thanks for your code!