无法正常运行ppocrv4_rec_server，其他可以正常运行

Open sunzx8 opened this issue 1 year ago • 1 comments

环境

【FastDeploy版本】：说明具体的版本，如fastdeploy-linux-gpu-0.8.0
- fastdeploy-gpu-python 1.0.7 fastdeploy-tools 0.0.5
运行指令：代码为示例代码：pipeline/multi_thread_process_ocr.py 运行方法： python multi_thread_process_ocr.py --det_model '/home/ubuntu/shawn/ocr-split/ocr_mix/models/ch_PP-OCRv4_det_infer' --cls_model 'ch_ppocr_mobile_v2.0_cls_infer' --rec_model '/home/ubuntu/shawn/ocr-split/ocr_mix/models/ch_PP-OCRv4_rec_server_infer' --rec_label_file '/home/ubuntu/shawn/FastDeploy/tutorials/multi_thread/python/pipeline/ppocr_keys_v1.txt' --image_path '/home/ubuntu/shawn/test_imgs/sda.jpg' --device gpu --use_multi_process True --process_num 3 --backend paddle

问题日志及出现问题的操作流程

附上详细的问题日志有助于快速定位分析
【模型跑不通】
- ppocrv4的server版本rec 无法运行，其他模型都可以正常运行，但是只要切换到server版本的识别模型就会报错

报错日志： multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/home/ubuntu/miniconda3/envs/ocr/lib/python3.10/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, **kwds)) File "/home/ubuntu/miniconda3/envs/ocr/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar return list(map(*args)) File "/home/ubuntu/shawn/FastDeploy/tutorials/multi_thread/python/pipeline/multi_thread_process_ocr.py", line 219, in process_predict result = ppocr_v3.predict(im) File "/home/ubuntu/miniconda3/envs/ocr/lib/python3.10/site-packages/fastdeploy/vision/ocr/ppocr/init.py", line 958, in predict return self.system_.predict(input_image) RuntimeError:

Compile Traceback (most recent call last): File "tools/export_model.py", line 288, in main() File "tools/export_model.py", line 284, in main model, arch_config, save_path, logger, input_shape=input_shape) File "tools/export_model.py", line 197, in export_single_model paddle.jit.save(model, save_path) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/jit.py", line 629, in wrapper func(layer, path, input_spec, **configs) File "/usr/local/lib/python3.7/dist-packages/decorator.py", line 232, in fun return caller(func, *(extras + args), **kw) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/wrapped_decorator.py", line 25, in impl return wrapped_func(*args, **kwargs) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/base.py", line 51, in impl return func(*args, **kwargs) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/jit.py", line 857, in save inner_input_spec, with_hook=with_hook) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 528, in concrete_program_specify_input_spec *desired_input_spec, with_hook=with_hook) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 436, in get_concrete_program concrete_program, partial_program_layer = self._program_cache[cache_key] File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 801, in getitem self._caches[item_id] = self._build_once(item) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 790, in _build_once **cache_key.kwargs) File "/usr/local/lib/python3.7/dist-packages/decorator.py", line 232, in fun return caller(func, *(extras + args), **kw) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/wrapped_decorator.py", line 25, in impl return wrapped_func(*args, **kwargs) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/base.py", line 51, in impl return func(*args, **kwargs) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 733, in from_func_spec outputs = static_func(*inputs) File "/tmp/tmp18grrv7q.py", line 28, in forward false_fn_1, (x,), (x,), (x,)) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 211, in convert_ifelse out = _run_py_ifelse(pred, true_fn, false_fn, true_args, false_args) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 257, in _run_py_ifelse return true_fn(*true_args) if pred else false_fn(*false_args) File "/workspace/gry/docker/code/luolixin/baidu/paddle/paddle-uapi/uapi/PaddleOCR/ppocr/modeling/architectures/base_model.py", line 86, in forward x = self.backbone(x) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call return self._dygraph_call_func(*inputs, **kwargs) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func outputs = self.forward(*inputs, **kwargs) File "/tmp/tmparp3ri44.py", line 45, in forward for_loop_condition_0, for_loop_body_0, [i, __for_loop_var_index_0, x]) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 45, in convert_while_loop loop_vars = _run_py_while(cond, body, loop_vars) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 59, in _run_py_while loop_vars = body(*loop_vars) File "/workspace/gry/docker/code/luolixin/baidu/paddle/paddle-uapi/uapi/PaddleOCR/ppocr/modeling/backbones/rec_hgnet.py", line 252, in forward x = stage(x) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call return self._dygraph_call_func(*inputs, **kwargs) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func outputs = self.forward(*inputs, **kwargs) File "/workspace/gry/docker/code/luolixin/baidu/paddle/paddle-uapi/uapi/PaddleOCR/ppocr/modeling/backbones/rec_hgnet.py", line 172, in forward x = self.blocks(x) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call return self._dygraph_call_func(*inputs, **kwargs) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func outputs = self.forward(*inputs, **kwargs) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/container.py", line 98, in forward input = layer(input) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call return self._dygraph_call_func(*inputs, **kwargs) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func outputs = self.forward(*inputs, **kwargs) File "/workspace/gry/docker/code/luolixin/baidu/paddle/paddle-uapi/uapi/PaddleOCR/ppocr/modeling/backbones/rec_hgnet.py", line 125, in forward x = self.att(x) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call return self._dygraph_call_func(*inputs, **kwargs) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func outputs = self.forward(*inputs, **kwargs) File "/workspace/gry/docker/code/luolixin/baidu/paddle/paddle-uapi/uapi/PaddleOCR/ppocr/modeling/backbones/rec_hgnet.py", line 76, in forward x = self.conv(x) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 930, in call return self._dygraph_call_func(*inputs, **kwargs) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func outputs = self.forward(*inputs, **kwargs) File "/usr/local/lib/python3.7/dist-packages/paddle/nn/layer/conv.py", line 678, in forward use_cudnn=self._use_cudnn) File "/usr/local/lib/python3.7/dist-packages/paddle/nn/functional/conv.py", line 169, in _conv_nd type=op_type, inputs=inputs, outputs=outputs, attrs=attrs) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/layer_helper.py", line 44, in append_op return self.main_program.current_block().append_op(*args, **kwargs) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/framework.py", line 3621, in append_op attrs=kwargs.get("attrs", None)) File "/usr/local/lib/python3.7/dist-packages/paddle/fluid/framework.py", line 2635, in init for frame in traceback.extract_stack():

C++ Traceback (most recent call last):

0 paddle::AnalysisPredictor::ZeroCopyRun() 1 paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, phi::Place const&) 2 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, phi::Place const&) const 3 paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, phi::Place const&, paddle::framework::RuntimeContext*) const 4 void phi::KernelImpl<void ()(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, paddle::optionalphi::DenseTensor const&, std::vector<int, std::allocator > const&, std::vector<int, std::allocator > const&, std::string const&, std::vector<int, std::allocator > const&, int, std::string const&, std::string const&, bool, std::vector<int, std::allocator > const&, int, phi::DenseTensor, std::vector<phi::DenseTensor*, std::allocatorphi::DenseTensor* >), &(void phi::fusion::ConvFusionKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, paddle::optionalphi::DenseTensor const&, std::vector<int, std::allocator > const&, std::vector<int, std::allocator > const&, std::string const&, std::vector<int, std::allocator > const&, int, std::string const&, std::string const&, bool, std::vector<int, std::allocator > const&, int, phi::DenseTensor*, std::vector<phi::DenseTensor*, std::allocatorphi::DenseTensor* >))>::KernelCallHelper<paddle::optionalphi::DenseTensor const&, std::vector<int, std::allocator > const&, std::vector<int, std::allocator > const&, std::string const&, std::vector<int, std::allocator > const&, int, std::string const&, std::string const&, bool, std::vector<int, std::allocator > const&, int, phi::DenseTensor*, std::vector<phi::DenseTensor*, std::allocatorphi::DenseTensor* >, phi::TypeTag >::Compute<1, 3, 0, 0, phi::GPUContext const, phi::DenseTensor const, phi::DenseTensor const, phi::DenseTensor const>(phi::KernelContext*, phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&) 5 void phi::fusion::ConvFusionKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, paddle::optionalphi::DenseTensor const&, std::vector<int, std::allocator > const&, std::vector<int, std::allocator > const&, std::string const&, std::vector<int, std::allocator > const&, int, std::string const&, std::string const&, bool, std::vector<int, std::allocator > const&, int, phi::DenseTensor*, std::vector<phi::DenseTensor*, std::allocatorphi::DenseTensor* >) 6 phi::DnnWorkspaceHandle::RunFunc(std::function<void (void*)> const&, unsigned long) 7 std::_Function_handler<void (void*), phi::fusion::ConvFusionKernel<float, phi::GPUContext>(phi::GPUContext const&, phi::DenseTensor const&, phi::DenseTensor const&, phi::DenseTensor const&, paddle::optionalphi::DenseTensor const&, std::vector<int, std::allocator > const&, std::vector<int, std::allocator > const&, std::string const&, std::vector<int, std::allocator > const&, int, std::string const&, std::string const&, bool, std::vector<int, std::allocator > const&, int, phi::DenseTensor*, std::vector<phi::DenseTensor*, std::allocatorphi::DenseTensor* >)::{lambda(void*)#4}>::_M_invoke(std::_Any_data const&, void*&&) 8 phi::enforce::EnforceNotMet::EnforceNotMet(phi::ErrorSummary const&, char const*, int) 9 phi::enforce::GetCurrentTraceBackStringabi:cxx11

Error Message Summary:

ExternalError: CUDNN error(9), CUDNN_STATUS_NOT_SUPPORTED. [Hint: Please search for the error code(9) on website (https://docs.nvidia.com/deeplearning/cudnn/api/index.html#cudnnStatus_t) to get Nvidia's official solution and advice about CUDNN Error.] (at /home/fastdeploy/develop/paddle_build/v0.0.0/Paddle/paddle/phi/kernels/fusion/gpu/conv_fusion_kernel.cu:611) [operator < conv2d_fusion > error] """

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/ubuntu/shawn/FastDeploy/tutorials/multi_thread/python/pipeline/multi_thread_process_ocr.py", line 250, in pool.map(process_predict, imgs_list) File "/home/ubuntu/miniconda3/envs/ocr/lib/python3.10/multiprocessing/pool.py", line 367, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/home/ubuntu/miniconda3/envs/ocr/lib/python3.10/multiprocessing/pool.py", line 774, in get raise self._value RuntimeError:

C++ Traceback (most recent call last):

Error Message Summary:

Nov 18 '24 05:11 sunzx8

重装cudnn v8.6.0，包治病

Dec 02 '24 07:12 Alex37882388