昇腾310P3测试demo运行后报错并卡住

[ly303550688]

• 版本、预测库信息：    1）Paddle Lite 版本：最新版本编译    2）Host 环境：容器，镜像registry.baidubce.com/device/paddle-npu:cann80RC1-ubuntu20-aarch64-gcc84-py39    3）运行设备环境：鲲鹏920+310P3
• 预测信息    1）预测 API：官方测试demo    2）预测选项信息：arm+ascend [ERROR] TBE(1094707,python3):2024-10-13-23:45:48.186.394 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:63][repository_manager] Subprocess[task_distribute] raise error[] [ERROR] TBE(1094711,python3):2024-10-13-23:45:48.186.430 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:63][repository_manager] Subprocess[task_distribute] raise error[] [ERROR] TBE(1094706,python3):2024-10-13-23:45:48.186.412 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:63][repository_manager] Subprocess[task_distribute] raise error[] [ERROR] TBE(1094710,python3):2024-10-13-23:45:48.186.430 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:63][repository_manager] Subprocess[task_distribute] raise error[] [ERROR] TBE(1094705,python3):2024-10-13-23:45:48.186.424 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:63][repository_manager] Subprocess[task_distribute] raise error[] [ERROR] TBE(1094704,python3):2024-10-13-23:45:48.186.437 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:63][repository_manager] Subprocess[task_distribute] raise error[] [ERROR] TBE(1094708,python3):2024-10-13-23:45:48.186.412 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:63][repository_manager] Subprocess[task_distribute] raise error[] [ERROR] TBE(1094709,python3):2024-10-13-23:45:48.186.432 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:63][repository_manager] Subprocess[task_distribute] raise error[] Process ForkServerPoolWorker-3: Process ForkServerPoolWorker-9: Process ForkServerPoolWorker-6: Process ForkServerPoolWorker-7: Process ForkServerPoolWorker-2: Process ForkServerPoolWorker-4: Process ForkServerPoolWorker-5: Process ForkServerPoolWorker-8: Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): Traceback (most recent call last): File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) Traceback (most recent call last): File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) Traceback (most recent call last): Traceback (most recent call last): File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) BrokenPipeError: [Errno 32] Broken pipe File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) Traceback (most recent call last): File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf)

During handling of the above exception, another exception occurred:

File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe BrokenPipeError: [Errno 32] Broken pipe File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 131, in worker put((job, i, result)) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) Traceback (most recent call last):

During handling of the above exception, another exception occurred:

BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) Traceback (most recent call last): BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run()

During handling of the above exception, another exception occurred:

BrokenPipeError: [Errno 32] Broken pipe File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size])

During handling of the above exception, another exception occurred:

File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) Traceback (most recent call last): File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() Traceback (most recent call last): File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run()

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() Traceback (most recent call last): File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) BrokenPipeError: [Errno 32] Broken pipe File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run()

During handling of the above exception, another exception occurred:

File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) Traceback (most recent call last): File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) BrokenPipeError: [Errno 32] Broken pipe File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) BrokenPipeError: [Errno 32] Broken pipe File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) BrokenPipeError: [Errno 32] Broken pipe BrokenPipeError: [Errno 32] Broken pipe File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 136, in worker put((job, i, (False, wrapped))) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put self._writer.send_bytes(obj) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes self._send_bytes(m[offset:offset + size]) BrokenPipeError: [Errno 32] Broken pipe File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes self._send(header + buf) File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe /root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 41 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d '

[ly303550688]

https://github.com/PaddlePaddle/Paddle-Lite/issues/10131 类似的问题

[ly303550688]

./run.sh mobilenet_v1_fp32_224 imagenet_224.txt test linux arm64 huawei_ascend_npu HUAWEI_ASCEND_NPU_SELECTED_DEVICE_IDS=0 cache 使用缓存方式，第二次没问题了，第一次会卡死

[MuShangCC]

现在使用缓存方式之后是没有报错吗

./run.sh mobilenet_v1_fp32_224 imagenet_224.txt test linux arm64 huawei_ascend_npu HUAWEI_ASCEND_NPU_SELECTED_DEVICE_IDS=0 cache 使用缓存方式，第二次没问题了，第一次会卡死

[ly303550688]

使用缓存第一次会卡死，生成缓存之后再重启程序就没问题

[MuShangCC]

您好，如果没有进一步的问题，我先把这个issue关闭了。如果还有问题，您随时可以 Reopen。

[hellobug-fei]

paddlelite3.12版本，昇腾310P3运行问题依然存在，模型预测为动态输入情况，缓存中不存在的shape依然会卡死，输入与缓存中相同的shape可以运行，希望官方定位一下原因