kie gpu推理出错CUDA error(9), invalid configuration argument.

[ThinkSYR]

• 系统环境/System Environment：linux Ubuntu 16.04 2080ti 11G显存
• 版本号/Version：Paddle：2.2.2.post101 PaddleOCR：latest(v2.6) 问题相关组件/Related components：
• 运行指令/Command Code：

python ./tools/infer_kie_token_ser_re.py \
  -c configs/kie/vi_layoutxlm/re_vi_layoutxlm_xfund_zh.yml \
  -o Architecture.Backbone.checkpoints=./pretrained_model/re_vi_layoutxlm_xfund_pretrained/best_accuracy \
  Global.infer_img=./train_data/XFUND/zh_val/image/zh_val_42.jpg \
  -c_ser configs/kie/vi_layoutxlm/ser_vi_layoutxlm_xfund_zh.yml \
  -o_ser Architecture.Backbone.checkpoints=./pretrained_model/ser_vi_layoutxlm_xfund_pretrained/best_accuracy

• 完整报错/Complete Error Message：

[2022/11/03 17:22:41] ppocr INFO:         shuffle : True
[2022/11/03 17:22:41] ppocr INFO: train with paddle 2.2.2 and device CUDAPlace(0)
W1103 17:22:41.598771 21757 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 7.5, Driver API Version: 10.1, Runtime API Version: 10.1
W1103 17:22:41.600710 21757 device_context.cc:465] device: 0, cuDNN Version: 7.6.
[2022/11/03 17:22:44] ppocr INFO: resume from ./pretrained_model/ser_vi_layoutxlm_xfund_pretrained/best_accuracy
[2022-11-03 17:22:45,122] [    INFO] - Already cached /home/imcs/.paddlenlp/models/layoutxlm-base-uncased/sentencepiece.bpe.model
[2022-11-03 17:22:45,466] [    INFO] - tokenizer config file saved in /home/imcs/.paddlenlp/models/layoutxlm-base-uncased/tokenizer_config.json
[2022-11-03 17:22:45,466] [    INFO] - Special tokens file saved in /home/imcs/.paddlenlp/models/layoutxlm-base-uncased/special_tokens_map.json
[2022/11/03 17:22:46] ppocr INFO: resume from ./pretrained_model/re_vi_layoutxlm_xfund_pretrained/best_accuracy
Corrupt JPEG data: premature end of data segment
Traceback (most recent call last):
  File "./tools/infer_kie_token_ser_re.py", line 217, in <module>
    result = ser_re_engine(data)
  File "./tools/infer_kie_token_ser_re.py", line 151, in __call__
    preds = self.model(re_input)
  File "/home/imcs/anaconda3/envs/ppocr/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 917, in __call__
    return self._dygraph_call_func(*inputs, **kwargs)
  File "/home/imcs/anaconda3/envs/ppocr/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 907, in _dygraph_call_func
    outputs = self.forward(*inputs, **kwargs)
  File "/data/hsy/PPOCREXP/PaddleOCR/ppocr/modeling/architectures/base_model.py", line 86, in forward
    x = self.backbone(x)
  File "/home/imcs/anaconda3/envs/ppocr/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 917, in __call__
    return self._dygraph_call_func(*inputs, **kwargs)
  File "/home/imcs/anaconda3/envs/ppocr/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 907, in _dygraph_call_func
    outputs = self.forward(*inputs, **kwargs)
  File "/data/hsy/PPOCREXP/PaddleOCR/ppocr/modeling/backbones/vqa_layoutlm.py", line 237, in forward
    relations=relations)
  File "/home/imcs/anaconda3/envs/ppocr/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 917, in __call__
    return self._dygraph_call_func(*inputs, **kwargs)
  File "/home/imcs/anaconda3/envs/ppocr/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 907, in _dygraph_call_func
    outputs = self.forward(*inputs, **kwargs)
  File "/home/imcs/anaconda3/envs/ppocr/lib/python3.7/site-packages/paddlenlp/transformers/layoutxlm/modeling.py", line 1559, in forward
    relations)
  File "/home/imcs/anaconda3/envs/ppocr/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 917, in __call__
    return self._dygraph_call_func(*inputs, **kwargs)
  File "/home/imcs/anaconda3/envs/ppocr/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 907, in _dygraph_call_func
    outputs = self.forward(*inputs, **kwargs)
  File "/home/imcs/anaconda3/envs/ppocr/lib/python3.7/site-packages/paddlenlp/transformers/layoutxlm/modeling.py", line 1464, in forward
    loss += self.loss_fct(logits, relation_labels)
  File "/home/imcs/anaconda3/envs/ppocr/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 917, in __call__
    return self._dygraph_call_func(*inputs, **kwargs)
  File "/home/imcs/anaconda3/envs/ppocr/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 907, in _dygraph_call_func
    outputs = self.forward(*inputs, **kwargs)
  File "/home/imcs/anaconda3/envs/ppocr/lib/python3.7/site-packages/paddle/nn/layer/loss.py", line 406, in forward
    name=self.name)
  File "/home/imcs/anaconda3/envs/ppocr/lib/python3.7/site-packages/paddle/nn/functional/loss.py", line 1755, in cross_entropy
    return _C_ops.mean(out)
OSError: (External) CUDA error(9), invalid configuration argument. 
  [Hint: 'cudaErrorInvalidConfiguration'. This indicates that a kernel launch is requesting resources that can never be satisfied by the current device. Requestingmore shared memory per block than the device supports will trigger this error, as will requesting too many threads or blocks.See cudaDeviceProp for more device limitations.] (at /paddle/paddle/fluid/operators/mean_op.cu:75)
  [operator < mean > error]

• 其他： 我在issue中找到了这个 #7739 ，但是他的报错信息跟我的还不太一样，我不确定是cuda版本的问题还是显存不足的问题，推理过程中需要很大显存吗？

[andyjiang1116]

显存多大呢，或者试试cpu推理，另外推荐使用paddle2.3版本

[ThinkSYR]

显存多大呢，或者试试cpu推理，另外推荐使用paddle2.3版本

显存11G的，cpu推理我试了demo是没问题的，是版本的问题吗，paddle2.3我之后试一下

[andyjiang1116]

试试最新版2.3吧

[ThinkSYR]

版本更新到2.3.2.post101就没有该问题了，谢谢