PaddleOCR 运行ppstructure demo 报错 (External) CUDA error(700), an illegal memory access was encountered.

运行ppstructure demo 报错 (External) CUDA error(700), an illegal memory access was encountered.

Open NBd-hub opened this issue 1 year ago • 2 comments

请提供下述完整信息以便快速定位问题/Please provide the following information to quickly locate the problem

系统环境/System Environment：unbuntu 18.04 docker cuda11.2+cudnn8
版本号/Version：Paddle： PaddleOCR：问题相关组件/Related components：paddle-gpu 2.3.1
运行指令/Command Code：python3 /home/paddle_new/PaddleOCR-release-2.5/ppstructure/predict_system.py
完整报错/Complete Error Message：OSError: In user code:

File "tools/export_model.py", line 107, in main() File "tools/export_model.py", line 103, in main run(FLAGS, cfg) File "tools/export_model.py", line 69, in run trainer.export(FLAGS.output_dir) File "/Users/anmengtao/Desktop/model_ED/PaddleDetection-release-2.0/ppdet/engine/trainer.py", line 460, in export input_spec, static_model.forward.main_program, File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 534, in main_program concrete_program = self.concrete_program File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 454, in concrete_program return self.concrete_program_specify_input_spec(input_spec=None) File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 487, in concrete_program_specify_input_spec concrete_program, _ = self.get_concrete_program( File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 402, in get_concrete_program concrete_program, partial_program_layer = self._program_cache[cache_key] File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 711, in getitem self._caches[item] = self._build_once(item) File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 698, in _build_once concrete_program = ConcreteProgram.from_func_spec( File "", line 2, in from_func_spec

File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/wrapped_decorator.py", line 25, in impl return wrapped_func(*args, **kwargs) File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/dygraph/base.py", line 39, in impl return func(*args, **kwargs) File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/dygraph/dygraph_to_static/program_translator.py", line 652, in from_func_spec outputs = static_func(*inputs) File "/var/folders/0t/0lmgjgfj7012tggtgbwtqby80000l2/T/tmpk8piw4sd.py", line 26, in forward out = paddle.jit.dy2static.convert_ifelse(self.training, true_fn_1, File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 210, in convert_ifelse return _run_py_ifelse(pred, true_fn, false_fn, true_args, false_args) File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 235, in _run_py_ifelse return true_fn(*true_args) if pred else false_fn(*false_args) File "/Users/anmengtao/Desktop/model_ED/PaddleDetection-release-2.0/ppdet/modeling/architectures/meta_arch.py", line 29, in forward out = self.get_pred() File "/Users/anmengtao/Desktop/model_ED/PaddleDetection-release-2.0/ppdet/modeling/architectures/yolo.py", line 75, in get_pred bbox_pred, bbox_num = self._forward() File "/Users/anmengtao/Desktop/model_ED/PaddleDetection-release-2.0/ppdet/modeling/architectures/yolo.py", line 59, in _forward body_feats = self.backbone(self.inputs) File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py", line 891, in call outputs = self.forward(*inputs, **kwargs) File "/var/folders/0t/0lmgjgfj7012tggtgbwtqby80000l2/T/tmp072_tt0k.py", line 40, in forward [idx, __for_loop_var_index_0, x] = paddle.jit.dy2static.convert_while_loop( File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 44, in convert_while_loop loop_vars = _run_py_while(cond, body, loop_vars) File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 58, in _run_py_while loop_vars = body(*loop_vars) File "/Users/anmengtao/Desktop/model_ED/PaddleDetection-release-2.0/ppdet/modeling/backbones/resnet.py", line 577, in forward x = stage(x) File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py", line 891, in call outputs = self.forward(*inputs, **kwargs) File "/var/folders/0t/0lmgjgfj7012tggtgbwtqby80000l2/T/tmpqo4t8ryz.py", line 18, in forward ] = paddle.jit.dy2static.convert_while_loop(for_loop_condition_1, File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 44, in convert_while_loop loop_vars = _run_py_while(cond, body, loop_vars) File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 58, in _run_py_while loop_vars = body(*loop_vars) File "/Users/anmengtao/Desktop/model_ED/PaddleDetection-release-2.0/ppdet/modeling/backbones/resnet.py", line 427, in forward block_out = block(block_out) File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py", line 891, in call outputs = self.forward(*inputs, **kwargs) File "/Users/anmengtao/Desktop/model_ED/PaddleDetection-release-2.0/ppdet/modeling/backbones/resnet.py", line 365, in forward out = self.branch2b(out) File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py", line 891, in call outputs = self.forward(*inputs, **kwargs) File "/var/folders/0t/0lmgjgfj7012tggtgbwtqby80000l2/T/tmph9feeafk.py", line 20, in forward out = paddle.jit.dy2static.convert_ifelse(paddle.jit.dy2static. File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 210, in convert_ifelse return _run_py_ifelse(pred, true_fn, false_fn, true_args, false_args) File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/dygraph/dygraph_to_static/convert_operators.py", line 235, in _run_py_ifelse return true_fn(*true_args) if pred else false_fn(*false_args) File "/Users/anmengtao/Desktop/model_ED/PaddleDetection-release-2.0/ppdet/modeling/backbones/resnet.py", line 130, in forward out = self.conv(inputs, offset, mask=mask) File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py", line 891, in call outputs = self.forward(*inputs, **kwargs) File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/vision/ops.py", line 765, in forward out = deform_conv2d( File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/vision/ops.py", line 578, in deform_conv2d helper.append_op( File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/layer_helper.py", line 43, in append_op return self.main_program.current_block().append_op(*args, **kwargs) File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/framework.py", line 3012, in append_op op = Operator( File "/Users/anmengtao/opt/anaconda3/lib/python3.8/site-packages/paddle/fluid/framework.py", line 2102, in init for frame in traceback.extract_stack():

ExternalError: CUBLAS error(1). [Hint: 'CUBLAS_STATUS_NOT_INITIALIZED'. The cuBLAS library was not initialized. This is usually caused by the lack of a prior cublasCreate() call, an error in the CUDA Runtime API called by the cuBLAS routine, or an error in the hardware setup. To correct: call cublasCreate() prior to the function call; and check that the hardware, an appropriate version of the driver, and the cuBLAS library are correctly installed. ] (at /paddle/paddle/phi/backends/gpu/gpu_resources.cc:140) [operator < deformable_conv > error] terminate called after throwing an instance of 'phi::enforce::EnforceNotMet' what(): (External) CUDA error(700), an illegal memory access was encountered. [Hint: 'cudaErrorIllegalAddress'. The device encountered a load or store instruction on an invalid memory address. This leaves the process in an inconsistentstate and any further CUDA work will return the same error. To continue using CUDA, the process must be terminated and relaunched. ] (at /paddle/paddle/fluid/platform/device/gpu/gpu_info.cc:235)

C++ Traceback (most recent call last):

0 paddle::AnalysisPredictor::~AnalysisPredictor() 1 paddle::AnalysisPredictor::~AnalysisPredictor() 2 paddle::memory::allocation::StreamSafeCUDAAllocator::ReleaseImpl(phi::Place const&) 3 paddle::memory::allocation::AutoGrowthBestFitAllocator::FreeIdleChunks() 4 paddle::memory::allocation::CUDAAllocator::FreeImpl(phi::Allocation*) 5 paddle::platform::RecordedGpuMallocHelper::Free(void*, unsigned long)

Error Message Summary:

FatalError: Process abort signal is detected by the operating system. [TimeInfo: *** Aborted at 1658816823 (unix time) try "date -d @1658816823" if you are using GNU date ***] [SignalInfo: *** SIGABRT (@0xe85) received by PID 3717 (TID 0x7f2520293740) from PID 3717 ***]