PaddlePaddle
PaddleOCR-VL本地运行失败AssertionError: Variable dtype not match
🔎 Search before asking
- [x] I have searched the PaddleOCR Docs and found no similar bug report.
- [x] I have searched the PaddleOCR Issues and found no similar bug report.
- [x] I have searched the PaddleOCR Discussions and found no similar bug report.
🐛 Bug (问题描述)
我采用官方的python脚本测试时报错如下：
Traceback (most recent call last): File "/paddle/demo_paddlevl.py", line 3, in <module> pipeline = PaddleOCRVL() File "/usr/local/lib/python3.10/dist-packages/paddleocr/_pipelines/paddleocr_vl.py", line 63, in __init__ super().__init__(**kwargs) File "/usr/local/lib/python3.10/dist-packages/paddleocr/_pipelines/base.py", line 67, in __init__ self.paddlex_pipeline = self._create_paddlex_pipeline() File "/usr/local/lib/python3.10/dist-packages/paddleocr/_pipelines/base.py", line 105, in _create_paddlex_pipeline return create_pipeline(config=self._merged_paddlex_config, **kwargs) File "/usr/local/lib/python3.10/dist-packages/paddlex/inference/pipelines/__init__.py", line 167, in create_pipeline pipeline = BasePipeline.get(pipeline_name)( File "/usr/local/lib/python3.10/dist-packages/paddlex/utils/deps.py", line 206, in _wrapper return old_init_func(self, *args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/paddlex/inference/pipelines/_parallel.py", line 103, in __init__ self._pipeline = self._create_internal_pipeline(config, self.device) File "/usr/local/lib/python3.10/dist-packages/paddlex/inference/pipelines/_parallel.py", line 158, in _create_internal_pipeline return self._pipeline_cls( File "/usr/local/lib/python3.10/dist-packages/paddlex/inference/pipelines/paddleocr_vl/pipeline.py", line 127, in __init__ self.vl_rec_model = self.create_model(vl_rec_config) File "/usr/local/lib/python3.10/dist-packages/paddlex/inference/pipelines/base.py", line 106, in create_model model = create_predictor( File "/usr/local/lib/python3.10/dist-packages/paddlex/inference/models/__init__.py", line 87, in create_predictor return BasePredictor.get(model_name)( File "/usr/local/lib/python3.10/dist-packages/paddlex/inference/models/doc_vlm/predictor.py", line 66, in __init__ self.infer, self.processor = self._build(**kwargs) File "/usr/local/lib/python3.10/dist-packages/paddlex/inference/models/doc_vlm/predictor.py", line 152, in _build model = PaddleOCRVLForConditionalGeneration.from_pretrained( File "/usr/local/lib/python3.10/dist-packages/paddlex/inference/models/common/vlm/transformers/model_utils.py", line 1986, in from_pretrained cls._load_pretrained_model( File "/usr/local/lib/python3.10/dist-packages/paddlex/inference/models/common/vlm/transformers/model_utils.py", line 1543, in _load_pretrained_model error_msgs = _load_state_dict_into_model( File "/usr/local/lib/python3.10/dist-packages/paddlex/inference/models/common/vlm/transformers/model_utils.py", line 299, in _load_state_dict_into_model model_to_load.set_hf_state_dict(state_dict) File "/usr/local/lib/python3.10/dist-packages/paddlex/inference/models/doc_vlm/modeling/paddleocr_vl/_paddleocr_vl.py", line 626, in set_hf_state_dict return self.set_state_dict(state_dict, *args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/paddle/base/framework.py", line 794, in wrapper return func(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/paddle/nn/layer/layers.py", line 2280, in set_state_dict param.set_value(state) File "/usr/local/lib/python3.10/dist-packages/paddle/base/wrapped_decorator.py", line 50, in wrapper return decorated(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/paddle/base/framework.py", line 722, in __impl__ return func(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/paddle/base/dygraph/tensor_patch_methods.py", line 237, in set_value assert self.dtype == dtype, ( AssertionError: Variable dtype not match, Variable [ linear_164.w_0 ] need tensor with dtype paddle.float32 but load tensor with dtype paddle.bfloat16
使用的官方python脚本如下： `from paddleocr import PaddleOCRVL import paddle pipeline = PaddleOCRVL()
后续推理代码
output = pipeline.predict("https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/paddleocr_vl_demo.png") for res in output: res.print()`
此外我运行CLI指令也一样报错
paddleocr doc_parser -i https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/paddleocr_vl_demo.png
请问需要怎样做呢？我尝试强制加载模型参数改成float32也还是报错
🏃♂️ Environment (运行环境)
使用的ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlepaddle/paddle:3.2.0-gpu-cuda11.8-cudnn8.9镜像构建的容器，参考官方指示在容器进行如下操作
python -m pip install -U "paddleocr[doc-parser]" python -m pip install https://paddle-whl.bj.bcebos.com/nightly/cu126/safetensors/safetensors-0.6.2.dev0-cp38-abi3-linux_x86_64.whl
CUDA 12.4
🌰 Minimal Reproducible Example (最小可复现问题的Demo)
paddleocr doc_parser -i https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/paddleocr_vl_demo.png
paddleocr doc_parser -i https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/paddleocr_vl_demo.png --device gpu:0提示：lib\site-packages\paddlex\inference\models\common\vlm\transformers\model_utils.py", line 254, in load_state_dict with safe_open(checkpoint_file, framework="paddle") as f: safetensors_rust.SafetensorError: framework paddle is invalid
我也遇到了类似的问题。
安装流程
参考：https://www.paddleocr.ai/latest/version3.x/pipeline_usage/PaddleOCR-VL.html#1
python -m pip install paddlepaddle-gpu==3.2.0 -i https://www.paddlepaddle.org.cn/packages/stable/cu126/
python -m pip install -U "paddleocr[doc-parser]"
python -m pip install https://paddle-whl.bj.bcebos.com/nightly/cu126/safetensors/safetensors-0.6.2.dev0-cp38-abi3-linux_x86_64.whl
gpu nvidia-T4
nvidia-smi
Fri Oct 17 20:49:25 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07 Driver Version: 550.90.07 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 Tesla T4 On | 00000000:00:09.0 Off | 0 |
| N/A 30C P8 9W / 70W | 1MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
运行报错:
运行paddleocr doc_parser -i https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/paddleocr_vl_demo.png报错：
/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddle/utils/cpp_extension/extension_utils.py:718: UserWarning: No ccache found. Please be aware that recompiling all source files may be required. You can download and install ccache from: https://github.com/ccache/ccache/blob/master/doc/INSTALL.md
warnings.warn(warning_message)
Creating model: ('PP-DocLayoutV2', None)
Model files already exist. Using cached files. To redownload, please delete the directory manually: `/.paddlex/official_models/PP-DocLayoutV2`.
Creating model: ('PaddleOCR-VL-0.9B', None)
Model files already exist. Using cached files. To redownload, please delete the directory manually: `/.paddlex/official_models/PaddleOCR-VL`.
Loading configuration file /.paddlex/official_models/PaddleOCR-VL/PaddleOCR-VL-0.9B/config.json
Loading weights file /.paddlex/official_models/PaddleOCR-VL/PaddleOCR-VL-0.9B/model.safetensors
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddle/utils/decorator_utils.py:420: Warning:
Non compatible API. Please refer to https://www.paddlepaddle.org.cn/documentation/docs/en/develop/guides/model_convert/convert_from_pytorch/api_difference/torch/torch.split.html first.
warnings.warn(
Loaded weights file from disk, setting weights to model.
Traceback (most recent call last):
File "/miniconda3/envs/paddleocr/bin/paddleocr", line 7, in <module>
sys.exit(console_entry())
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddleocr/__main__.py", line 26, in console_entry
main()
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddleocr/_cli.py", line 192, in main
_execute(args)
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddleocr/_cli.py", line 181, in _execute
args.executor(args)
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddleocr/_pipelines/paddleocr_vl.py", line 363, in execute_with_args
perform_simple_inference(
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddleocr/_utils/cli.py", line 62, in perform_simple_inference
wrapper = wrapper_cls(**init_params)
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddleocr/_pipelines/paddleocr_vl.py", line 63, in __init__
super().__init__(**kwargs)
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddleocr/_pipelines/base.py", line 67, in __init__
self.paddlex_pipeline = self._create_paddlex_pipeline()
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddleocr/_pipelines/base.py", line 105, in _create_paddlex_pipeline
return create_pipeline(config=self._merged_paddlex_config, **kwargs)
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/inference/pipelines/__init__.py", line 167, in create_pipeline
pipeline = BasePipeline.get(pipeline_name)(
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/utils/deps.py", line 206, in _wrapper
return old_init_func(self, *args, **kwargs)
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/inference/pipelines/_parallel.py", line 103, in __init__
self._pipeline = self._create_internal_pipeline(config, self.device)
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/inference/pipelines/_parallel.py", line 158, in _create_internal_pipeline
return self._pipeline_cls(
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/inference/pipelines/paddleocr_vl/pipeline.py", line 127, in __init__
self.vl_rec_model = self.create_model(vl_rec_config)
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/inference/pipelines/base.py", line 106, in create_model
model = create_predictor(
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/inference/models/__init__.py", line 87, in create_predictor
return BasePredictor.get(model_name)(
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/inference/models/doc_vlm/predictor.py", line 66, in __init__
self.infer, self.processor = self._build(**kwargs)
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/inference/models/doc_vlm/predictor.py", line 152, in _build
model = PaddleOCRVLForConditionalGeneration.from_pretrained(
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/inference/models/common/vlm/transformers/model_utils.py", line 1986, in from_pretrained
cls._load_pretrained_model(
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/inference/models/common/vlm/transformers/model_utils.py", line 1543, in _load_pretrained_model
error_msgs = _load_state_dict_into_model(
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/inference/models/common/vlm/transformers/model_utils.py", line 299, in _load_state_dict_into_model
model_to_load.set_hf_state_dict(state_dict)
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/inference/models/doc_vlm/modeling/paddleocr_vl/_paddleocr_vl.py", line 626, in set_hf_state_dict
return self.set_state_dict(state_dict, *args, **kwargs)
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddle/base/framework.py", line 794, in wrapper
return func(*args, **kwargs)
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 2280, in set_state_dict
param.set_value(state)
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddle/base/wrapped_decorator.py", line 50, in wrapper
return decorated(*args, **kwargs)
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddle/base/framework.py", line 722, in __impl__
return func(*args, **kwargs)
File "/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddle/base/dygraph/tensor_patch_methods.py", line 237, in set_value
assert self.dtype == dtype, (
AssertionError: Variable dtype not match, Variable [ linear_164.w_0 ] need tensor with dtype paddle.float32 but load tensor with dtype paddle.bfloat16
我是完全按照文档操作的。env 也是全新创建的。 使用如下的 python 代码执行时，也是一样报错：
from paddleocr import PaddleOCRVL
pipeline = PaddleOCRVL()
output = pipeline.predict("https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/paddleocr_vl_demo.png")
for res in output:
res.print()
res.save_to_json(save_path="output")
res.save_to_markdown(save_path="output")
首先感谢您的使用，目前 Compute Capacity <= 7.5 的卡例如 T4、V100 不支持原生推理，可以参考文档使用 vLLM部署，但 V100、T4 可能会出现服务不稳定的情况，还是不太推荐使用，看能不能换别的卡？如3060，4070，A卡等等。 我们后续也会根据大家的使用情况看能不能适配更多硬件。
指定 device cpu 也是这个错误
$ paddleocr doc_parser -i https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/paddleocr_vl_demo.png --device cpu
Creating model: ('PP-DocLayoutV2', None)
Model files already exist. Using cached files. To redownload, please delete the directory manually: `/home/iwater/.paddlex/official_models/PP-DocLayoutV2`.
/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddle/utils/cpp_extension/extension_utils.py:718: UserWarning: No ccache found. Please be aware that recompiling all source files may be required. You can download and install ccache from: https://github.com/ccache/ccache/blob/master/doc/INSTALL.md
warnings.warn(warning_message)
Creating model: ('PaddleOCR-VL-0.9B', None)
Model files already exist. Using cached files. To redownload, please delete the directory manually: `/home/iwater/.paddlex/official_models/PaddleOCR-VL`.
Loading configuration file /home/iwater/.paddlex/official_models/PaddleOCR-VL/PaddleOCR-VL-0.9B/config.json
Loading weights file /home/iwater/.paddlex/official_models/PaddleOCR-VL/PaddleOCR-VL-0.9B/model.safetensors
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
use GQA - num_heads: 16- num_key_value_heads: 2
/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddle/utils/decorator_utils.py:420: Warning:
Non compatible API. Please refer to https://www.paddlepaddle.org.cn/documentation/docs/en/develop/guides/model_convert/convert_from_pytorch/api_difference/torch/torch.split.html first.
warnings.warn(
Loaded weights file from disk, setting weights to model.
Traceback (most recent call last):
File "/home/iwater/miniconda3/envs/paddleocr/bin/paddleocr", line 7, in <module>
sys.exit(console_entry())
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddleocr/__main__.py", line 26, in console_entry
main()
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddleocr/_cli.py", line 192, in main
_execute(args)
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddleocr/_cli.py", line 181, in _execute
args.executor(args)
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddleocr/_pipelines/paddleocr_vl.py", line 363, in execute_with_args
perform_simple_inference(
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddleocr/_utils/cli.py", line 62, in perform_simple_inference
wrapper = wrapper_cls(**init_params)
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddleocr/_pipelines/paddleocr_vl.py", line 63, in __init__
super().__init__(**kwargs)
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddleocr/_pipelines/base.py", line 67, in __init__
self.paddlex_pipeline = self._create_paddlex_pipeline()
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddleocr/_pipelines/base.py", line 105, in _create_paddlex_pipeline
return create_pipeline(config=self._merged_paddlex_config, **kwargs)
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/inference/pipelines/__init__.py", line 167, in create_pipeline
pipeline = BasePipeline.get(pipeline_name)(
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/utils/deps.py", line 206, in _wrapper
return old_init_func(self, *args, **kwargs)
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/inference/pipelines/_parallel.py", line 103, in __init__
self._pipeline = self._create_internal_pipeline(config, self.device)
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/inference/pipelines/_parallel.py", line 158, in _create_internal_pipeline
return self._pipeline_cls(
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/inference/pipelines/paddleocr_vl/pipeline.py", line 127, in __init__
self.vl_rec_model = self.create_model(vl_rec_config)
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/inference/pipelines/base.py", line 106, in create_model
model = create_predictor(
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/inference/models/__init__.py", line 87, in create_predictor
return BasePredictor.get(model_name)(
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/inference/models/doc_vlm/predictor.py", line 66, in __init__
self.infer, self.processor = self._build(**kwargs)
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/inference/models/doc_vlm/predictor.py", line 152, in _build
model = PaddleOCRVLForConditionalGeneration.from_pretrained(
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/inference/models/common/vlm/transformers/model_utils.py", line 1986, in from_pretrained
cls._load_pretrained_model(
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/inference/models/common/vlm/transformers/model_utils.py", line 1543, in _load_pretrained_model
error_msgs = _load_state_dict_into_model(
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/inference/models/common/vlm/transformers/model_utils.py", line 299, in _load_state_dict_into_model
model_to_load.set_hf_state_dict(state_dict)
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddlex/inference/models/doc_vlm/modeling/paddleocr_vl/_paddleocr_vl.py", line 626, in set_hf_state_dict
return self.set_state_dict(state_dict, *args, **kwargs)
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddle/base/framework.py", line 794, in wrapper
return func(*args, **kwargs)
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddle/nn/layer/layers.py", line 2280, in set_state_dict
param.set_value(state)
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddle/base/wrapped_decorator.py", line 50, in wrapper
return decorated(*args, **kwargs)
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddle/base/framework.py", line 722, in __impl__
return func(*args, **kwargs)
File "/home/iwater/miniconda3/envs/paddleocr/lib/python3.10/site-packages/paddle/base/dygraph/tensor_patch_methods.py", line 237, in set_value
assert self.dtype == dtype, (
AssertionError: Variable dtype not match, Variable [ linear_164.w_0 ] need tensor with dtype paddle.float32 but load tensor with dtype paddle.bfloat16
需要 Compute Capacity > 75 的 GPU 运行哈，T4、V100 这种 70，75 的也可以试试 vLLM 部署，不过他们在高吞吐下不太稳定。
A GPU with Compute Capacity > 75 is required to run this. GPUs with Compute Capacity > 75, such as T4 and V100, can also try vLLM deployment, but they are not very stable under high throughput.
So using CPU inference is impossible for this model?
首先感谢您的使用，目前 Compute Capacity <= 7.5 的卡例如 T4、V100 不支持原生推理，可以参考文档使用 vLLM部署，但 V100、T4 可能会出现服务不稳定的情况，还是不太推荐使用，看能不能换别的卡？如3060，4070，A卡等等。 我们后续也会根据大家的使用情况看能不能适配更多硬件。
Traceback (most recent call last):
File "/root/autodl-tmp/data/miniconda3/envs/paddle/bin/paddleocr", line 7, in
paddleocr doc_parser -i https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/paddleocr_vl_demo.png --device gpu:0提示：lib\site-packages\paddlex\inference\models\common\vlm\transformers\model_utils.py", line 254, in load_state_dict with safe_open(checkpoint_file, framework="paddle") as f: safetensors_rust.SafetensorError: framework paddle is invalid
建议使用更高版本的CUDA环境测试
Issue Confirmation and Workaround
I encountered the same AssertionError: Variable dtype not match error when attempting to run PaddleOCR-VL locally on Linux (CPU). The root cause appears to be a dtype incompatibility: the model weights from Hugging Face are stored in bfloat16, but PaddlePaddle's loading mechanism expects float32, leading to failure during model initialization.
Temporary Workaround
As a non-official fix, I modified the file paddlex/inference/models/common/vlm/transformers/model_utils.py in the _load_state_dict_into_model function to perform an on-the-fly dtype conversion.
Insert the following code snippet immediately before model_to_load.set_hf_state_dict(state_dict):
# Convert bfloat16 to float32 if needed to resolve dtype mismatch
for key, value in state_dict.items():
if hasattr(value, 'dtype') and value.dtype == paddle.bfloat16:
state_dict[key] = value.astype(paddle.float32)
This change allows the model to load successfully and process images without errors. Note: This may introduce minor performance overhead due to the conversion, but it resolves the immediate issue.
Recommendations Official Fix Needed: The PaddleOCR team should address this compatibility issue in a future release to handle dtype mismatches automatically during model loading. Testing: Verified on the demo image from the PaddleOCR-VL repository; output matches expectations post-fix. Environment Details PaddlePaddle: 3.2.0 (CPU) PaddleOCR: 3.3.0 Python: 3.13 OS: Fedora Linux 42 (Workstation Edition) Hardware: CPU (x64) This workaround is intended for development/testing purposes only and should not be used in production without further validation.
Thank you, @Plaster0833 , for providing the temporary workaround.
The root cause of this issue is that the released model uses bfloat16. Currently, GPUs that do not support bfloat16 are not supported.
We are actively working on a solution and expect to have it ready as soon as possible.
The issue has no response for a long time and will be closed. You can reopen or new another issue if are still confused.
From Bot