Serving
Serving copied to clipboard
PaddlePaddle
A10的GPU上运行模型，过一段时间就会报CUDA error(700), an illegal memory access was encountered，机器看不出任何问题
ERROR 2023-08-18 22:36:56,242 [operator.py:1079] [text_quality] failed to predict. (data_id=517045 log_id=517045) [text_quality|6] Failed to process(batch: [517045]): (External) CUDA error(700), an illegal memory access was encountered. [Hint: Please search for the error code(700) on website (https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1g3f51e3575c2178246db0a94a430e0038) to get Nvidia's official solution and advice about CUDA Error.] (at /paddle/paddle/phi/backends/gpu/cuda/cuda_info.cc:252) . Please check the input dict and checkout PipelineServingLogs/pipeline.log for more details. INFO 2023-08-18 22:36:56,242 [operator.py:1454] prometheus inf count +1 ERROR 2023-08-18 22:36:56,247 [dag.py:420] (data_id=517045 log_id=0) Failed to predict: [text_quality] failed to predict. (data_id=517045 log_id=517045) [text_quality|6] Failed to process(batch: [517045]): (External) CUDA error(700), an illegal memory access was encountered. [Hint: Please search for the error code(700) on website (https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1g3f51e3575c2178246db0a94a430e0038) to get Nvidia's official solution and advice about CUDA Error.] (at /paddle/paddle/phi/backends/gpu/cuda/cuda_info.cc:252) . Please check the input dict and checkout PipelineServingLogs/pipeline.log for more details
nvidia-smi +-----------------------------------------------------------------------------+ | NVIDIA-SMI 460.91.03 Driver Version: 460.91.03 CUDA Version: 11.2 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 A10 On | 00000000:5E:00.0 Off | 0 | | 0% 44C P0 57W / 150W | 6606MiB / 22731MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================|
没看出任何问题，
