depthai-python
depthai-python copied to clipboard
luxonis
OAK-1 MAX Log: 'Fatal error on MSS CPU: trap: 09, address: 802AD144' '0'
Unable to run YOLO ros example on the MAX-1?
[yolov4_spatial_node-1] [2023-03-18 21:54:48.006] [debug] Asset map dump: {"map":{"/node/1/__blob":{"alignment":64,"offset":0,"size":12172416}}}
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.169] [ColorCamera(0)] [warning] LCM48 supported resolutions: 4_K / 4000X3000 / 5312X6000. Defaulting to 4_K
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.169] [MonoCamera(2)] [error] Camera not detected on socket: 1
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.169] [MonoCamera(2)] [info] Using board socket: 1, id: 1
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.169] [MonoCamera(3)] [error] Camera not detected on socket: 2
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.170] [MonoCamera(3)] [info] Using board socket: 2, id: 2
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.173] [system] [info] SpatialLocationCalculator shave buffer size '29696'B
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [2.458] [system] [critical] Fatal error. Please report to developers. Log: 'Fatal error on MSS CPU: trap: 09, address: 802AD144' '0'
[yolov4_spatial_node-1] [2023-03-18 21:54:51.522] [debug] DataOutputQueue (detections) closed
[yolov4_spatial_node-1] [2023-03-18 21:54:51.522] [debug] DataOutputQueue (depth) closed
[yolov4_spatial_node-1] [2023-03-18 21:54:51.523] [debug] Device about to be closed...
[yolov4_spatial_node-1] [2023-03-18 21:54:51.524] [debug] DataOutputQueue (preview) closed
[yolov4_spatial_node-1] [2023-03-18 21:54:51.525] [debug] Timesync thread exception caught: Couldn't read data from stream: '__timesync' (X_LINK_ERROR)
[yolov4_spatial_node-1] [2023-03-18 21:54:51.525] [debug] Log thread exception caught: Couldn't read data from stream: '__log' (X_LINK_ERROR)
[yolov4_spatial_node-1] [2023-03-18 21:54:51.718] [debug] Watchdog thread exception caught: Couldn't write data to stream: '__watchdog' (X_LINK_ERROR)
[yolov4_spatial_node-1] [2023-03-18 21:54:52.174] [debug] XLinkResetRemote of linkId: (0)
[yolov4_spatial_node-1] [2023-03-18 21:54:52.466] [debug] Device closed, 942
[yolov4_spatial_node-1] terminate called after throwing an instance of 'dai::XLinkReadError'
[yolov4_spatial_node-1] what(): Couldn't read data from stream: '__rpc_main' (X_LINK_ERROR)
[ERROR] [yolov4_spatial_node-1]: process has died [pid 579125, exit code -6, cmd '/home/ubuntu/luxonis_oak_ws/install/depthai_examples/lib/depthai_examples/yolov4_spatial_node --ros-args --params-file /tmp/launch_params_yi0rp5n8 --params-file /tmp/launch_params_ofahoqpi --params-file /tmp/launch_params_nql3degb --params-file /tmp/launch_params_wng5cwr7 --params-file /tmp/launch_params_yi58p6s4 --params-file /tmp/launch_params_3bl7dpti'].
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.066] [system] [warning] PRINT:LeonCss: Calling IPC set assets
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.176] [system] [warning] PRINT:LeonCss: Asset: /node/1/__blob, pointer: 0x8abb5380, size: 12172416, alignment: 64
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.176] [ColorCamera(0)] [warning] LCM48 supported resolutions: 4_K / 4000X3000 / 5312X6000. Defaulting to 4_K
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.176] [MonoCamera(2)] [error] Camera not detected on socket: 1
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.177] [MonoCamera(2)] [info] Using board socket: 1, id: 1
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.177] [MonoCamera(3)] [error] Camera not detected on socket: 2
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.177] [MonoCamera(3)] [info] Using board socket: 2, id: 2
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.187] [system] [warning] PRINT:LeonMss: Calculating resources
[yolov4_spatial_node-1] Number of cmx slices available 16
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.181] [system] [info] SpatialLocationCalculator shave buffer size '29696'B
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.220] [system] [warning] PRINT:LeonMss: sippPalThreadCreate: Thread /SIPP created
[yolov4_spatial_node-1] Number of column of the first matrix should match with the number of rows of the second matrix 3x3 - 0
[yolov4_spatial_node-1] Unexpected trap ( 9) at address 0x802ad144
[yolov4_spatial_node-1] Bad trap, 0x9
[yolov4_spatial_node-1] PSR: 0xf34010c2 PC: 0x802ad148 NPC: 0x802ad14c TPC: 0x802ad144
[yolov4_spatial_node-1] G1: 0x00000000 G2: 0x00000000 G3: 0x00000000 G4: 0x00000000
[yolov4_spatial_node-1] G5: 0x00000000 G6: 0x806f43c0 G7: 0x00000000 I0: 0x00000000
[yolov4_spatial_node-1] I1: 0xaaaaa800 I2: 0x00000000 I3: 0x806f4390 I4: 0xaaaaaaab
[yolov4_spatial_node-1] I5: 0x82cc44c0 I6: 0x806f4328 I7: 0x8001d7e4 Y: 0x00000000
[yolov4_spatial_node-1]
[yolov4_spatial_node-1] L0: 0x806f4504 L1: 0x82cc44c0 L2: 0x82cc44e4 L3: 0x806f4914
[yolov4_spatial_node-1] L4: 0x806f492c L5: 0x806f4944 L6: 0x806f4994 L7: 0x806f49a8
[yolov4_spatial_node-1] I0: 0x806f44f8 I1: 0x806f44e0 I2: 0x806f43c0 I3: 0x82cc44e4
[yolov4_spatial_node-1] I4: 0x806f439c I5: 0xaaaaaaab I6: 0x806f43c0 I7: 0x8014f3fc
[yolov4_spatial_node-1] SRA: 0x806f4374 SA0:0x806f4388 SA1: 0x806f4864 SA2: 0x00000000
[yolov4_spatial_node-1] SA3: 0x00000000 SA4:0x00000000 SA5: 0x806f4504
[yolov4_spatial_node-1] Unexpected trap ( 9) at address 0x802ad144
[yolov4_spatial_node-1] Bad trap, 0x9
[yolov4_spatial_node-1] PSR: 0xf34010c2 PC: 0x802ad148 NPC: 0x802ad14c TPC: 0x802ad144
[yolov4_spatial_node-1] G1: 0x00000000 G2: 0x00000000 G3: 0x00000000 G4: 0x00000000
[yolov4_spatial_node-1] G5: 0x00000000 G6: 0x806f43c0 G7: 0x00000000 I0: 0x00000000
[yolov4_spatial_node-1] I1: 0xaaaaa800 I2: 0x00000000 I3: 0x806f4390 I4: 0xaaaaaaab
[yolov4_spatial_node-1] I5: 0x82cc44c0 I6: 0x806f4328 I7: 0x8001d7e4 Y: 0x00000000
[yolov4_spatial_node-1]
[yolov4_spatial_node-1] L0: 0x806f4504 L1: 0x82cc44c0 L2: 0x82cc44e4 L3: 0x806f4914
[yolov4_spatial_node-1] L4: 0x806f492c L5: 0x806f4944 L6: 0x806f4994 L7: 0x806f49a8
[yolov4_spatial_node-1] I0: 0x806f44f8 I1: 0x806f44e0 I2: 0x806f43c0 I3: 0x82cc44e4
[yolov4_spatial_node-1] I4: 0x806f439c I5: 0xaaaaaaab I6: 0x806f43c0 I7: 0x8014f3fc
[yolov4_spatial_node-1] SRA: 0x806f4374 SA0:0x806f4388 SA1: 0x806f4864 SA2: 0x00000000
[yolov4_spatial_node-1] SA3: 0x00000000 SA4:0x00000000 SA5: 0x806f4504
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.539] [system] [warning] PRINT:LeonCss: initial keepalive, countdown: 8
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [2.461] [system] [critical] Fatal error. Please report to developers. Log: 'Fatal error on MSS CPU: trap: 09, address: 802AD144' '0'
@MAVProxyUser
Do you mind running crash_dump.py example afterwards & send that over?
(WIP to have that done semi-automatically in the future)
Oh @themarpe you mean this? https://github.com/luxonis/depthai-docs-website/commit/8497ef71df7475de50f2f5cb8a2339bc0cfbbe8c
Compiling develop branch per these notes now: Will check back. https://docs.luxonis.com/en/latest/pages/troubleshooting/
@themarpe sorry brother... I can't get the develop branch to compile on my machine .
[2023-03-20 01:24:45.371] [debug] Library information - version: 2.20.2, commit: fc81e1b7d0c66813b9e75dbe83191d8c9e1db66e from 2023-03-19 20:09:09 +0100, build: 2023-03-20 04:46:23 +0000
[2023-03-20 01:24:45.377] [debug] Initialize - finished
[2023-03-20 01:24:45.655] [debug] Resources - Archive 'depthai-bootloader-fwp-0.0.24.tar.xz' open: 8ms, archive read: 273ms
Traceback (most recent call last):
File "/usr/bin/stubgen", line 11, in <module>
load_entry_point('mypy==0.761', 'console_scripts', 'stubgen')()
File "mypy/stubgen.py", line 1559, in main
File "mypy/stubgen.py", line 1452, in generate_stubs
File "mypy/stubgenc.py", line 51, in generate_stub_for_c_module
File "mypy/stubgenc.py", line 273, in generate_c_type_stub
File "mypy/stubgenc.py", line 221, in generate_c_property_stub
TypeError: str object expected; got None
[2023-03-20 01:24:46.258] [debug] Resources - Archive 'depthai-device-fwp-cbab456e3481d680f7f125fbc3b5ec0435323a83.tar.xz' open: 8ms, archive read: 877ms
make[2]: *** [CMakeFiles/depthai.dir/build.make:987: depthai.cpython-38-aarch64-linux-gnu.so] Error 1
make[2]: *** Deleting file 'depthai.cpython-38-aarch64-linux-gnu.so'
make[1]: *** [CMakeFiles/Makefile2:187: CMakeFiles/depthai.dir/all] Error 2
Ok I had to upgrade the mypy version
python3 -m pip install -U git+https://github.com/python/mypy.git
ok @themarpe after some fussing around I figured out how to get the right branches. I triggered the crash...
[yolov4_spatial_node-1] [1844301011A9460E00] [1.1] [1.129] [system] [warning] PRINT:LeonMss: sippPalThreadCreate: Thread /SIPP created
[yolov4_spatial_node-1] Number of column of the first matrix should match with the number of rows of the second matrix 3x3 - 0
[yolov4_spatial_node-1] Unexpected trap ( 9) at address 0x802936e8
[yolov4_spatial_node-1] Bad trap, 0x9
[yolov4_spatial_node-1] PSR: 0xf34010c2 PC: 0x802936ec NPC: 0x802936f0 TPC: 0x802936e8
[yolov4_spatial_node-1] G1: 0x00000000 G2: 0x00000000 G3: 0x00000000 G4: 0x00000000
[yolov4_spatial_node-1] G5: 0x00000000 G6: 0x806d7260 G7: 0x00000000 I0: 0x00000000
[yolov4_spatial_node-1] I1: 0xaaaaa800 I2: 0x00000000 I3: 0x806d7230 I4: 0xaaaaaaab
[yolov4_spatial_node-1] I5: 0x82ccce00 I6: 0x806d71c8 I7: 0x8001d568 Y: 0x00000000
[yolov4_spatial_node-1]
[yolov4_spatial_node-1] L0: 0x806d73b4 L1: 0x82ccce00 L2: 0x82ccce24 L3: 0x806d77c4
[yolov4_spatial_node-1] L4: 0x806d77dc L5: 0x806d77f4 L6: 0x806d7844 L7: 0x806d7858
[yolov4_spatial_node-1] I0: 0x806d73a8 I1: 0x806d7390 I2: 0x806d7260 I3: 0x82ccce24
[yolov4_spatial_node-1] I4: 0x806d723c I5: 0xaaaaaaab I6: 0x806d7260 I7: 0x8014cf14
[yolov4_spatial_node-1] SRA: 0x806d7214 SA0:0x806d7228 SA1: 0x806d7714 SA2: 0x00000000
[yolov4_spatial_node-1] SA3: 0x00000000 SA4:0x00000000 SA5: 0x806d73b4
[yolov4_spatial_node-1] Unexpected trap ( 9) at address 0x802936e8
[yolov4_spatial_node-1] Bad trap, 0x9
[yolov4_spatial_node-1] PSR: 0xf34010c2 PC: 0x802936ec NPC: 0x802936f0 TPC: 0x802936e8
[yolov4_spatial_node-1] G1: 0x00000000 G2: 0x00000000 G3: 0x00000000 G4: 0x00000000
[yolov4_spatial_node-1] G5: 0x00000000 G6: 0x806d7260 G7: 0x00000000 I0: 0x00000000
[yolov4_spatial_node-1] I1: 0xaaaaa800 I2: 0x00000000 I3: 0x806d7230 I4: 0xaaaaaaab
[yolov4_spatial_node-1] I5: 0x82ccce00 I6: 0x806d71c8 I7: 0x8001d568 Y: 0x00000000
[yolov4_spatial_node-1]
[yolov4_spatial_node-1] L0: 0x806d73b4 L1: 0x82ccce00 L2: 0x82ccce24 L3: 0x806d77c4
[yolov4_spatial_node-1] L4: 0x806d77dc L5: 0x806d77f4 L6: 0x806d7844 L7: 0x806d7858
[yolov4_spatial_node-1] I0: 0x806d73a8 I1: 0x806d7390 I2: 0x806d7260 I3: 0x82ccce24
[yolov4_spatial_node-1] I4: 0x806d723c I5: 0xaaaaaaab I6: 0x806d7260 I7: 0x8014cf14
[yolov4_spatial_node-1] SRA: 0x806d7214 SA0:0x806d7228 SA1: 0x806d7714 SA2: 0x00000000
[yolov4_spatial_node-1] SA3: 0x00000000 SA4:0x00000000 SA5: 0x806d73b4
[yolov4_spatial_node-1] [1844301011A9460E00] [1.1] [1.547] [system] [warning] PRINT:LeonCss: initial keepalive, countdown: 8
[yolov4_spatial_node-1] [1844301011A9460E00] [1.1] [2.401] [system] [critical] Fatal error. Please report to developers. Log: 'Fatal error on MSS CPU: trap: 09, address: 802936E8' '0'
but the crash_report script shows nothing.
kfinisterre@dev1:~/depthai-python$ python3 ./examples/CrashReport/crash_report.py
There was no crash dump found on your device!
@MAVProxyUser the example doesn't work, because spatial examples require stereo cameras, and stereo depth. OAK-1-MAX has only 1 camera.
@szabi-luxonis YOLO doesn't require stereo depth ... why should it here? It runs just fine via depthai_demo.py
This is literally off the same OAK-1-MAX camera. You can see there is no issue using YOLO with OAK-1-MAX, I'd have the expectation that the same detections would be returned by https://github.com/luxonis/depthai-ros/blob/humble/depthai_examples/launch/yolov4_publisher.launch.py
@MAVProxyUser you are running an example called: yolov4_spatial_node. Spatial node involves stereo depth.
Error logs also point to missing stereo cameras, as expected.
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.169] [MonoCamera(2)] [error] Camera not detected on socket: 1
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.169] [MonoCamera(2)] [info] Using board socket: 1, id: 1
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.169] [MonoCamera(3)] [error] Camera not detected on socket: 2
[yolov4_spatial_node-1] [1844301011A9460E00] [1.2.4] [1.170] [MonoCamera(3)] [info] Using board socket: 2, id: 2
@szabi-luxonis Is there a version of yolov4_publisher.launch.py that does not launch yolov4_spatial_node, or has a more intuitive flag to apply to the launch command-line so it starts with spatial disabled? There was seemingly nothing that indicated "stereo cameras" are "expected" until you pointed out that error buried in the logs. For me this is completely unexpected behavior for something that previously didn't indicate any need for a Stereo Cam. When I think YOLO, I don't at all think "depth is a requirement".
I had no clue it was dependent on spatial data until you just said so, and I subsequently looked at the code here: https://github.com/luxonis/depthai-ros/blob/humble/depthai_examples/launch/yolov4_publisher.launch.py#LL147C12-L147C19
yolov4_spatial_node = launch_ros.actions.Node(
package='depthai_examples', executable='yolov4_spatial_node',
output='screen',
parameters=[{'tf_prefix': tf_prefix},
{'camera_param_uri': camera_param_uri},
{'sync_nn': sync_nn},
{'nnName': nnName},
{'resourceBaseFolder': resourceBaseFolder},
{'monoResolution': monoResolution}])
As a new end user... can you help me out in getting a non-spatial example to run out of the box. I expected to be able to swap out my other Oak cameras in testing so I could compare performance. I have Oak-D-S2, Oak-D-Pro and two Oak-1 variants.
At a glance @szabi-luxonis there is no non spatial YOLO example included, only this? https://github.com/luxonis/depthai-ros/blob/humble/depthai_examples/src/yolov4_spatial_publisher.cpp
@szabi-luxonis do you have a suggestion as to how one would move forward getting an equivalent yolov4_publisher.launch.py that does not use spatial? Obviously several of your camera's are sold sans spatial options, and it would be nice to still have a YOLO example node for them.
@MAVProxyUser I would suggest opening a request on depthai-ros repository, and/or modifying the current example (you just need to remove extra code creating mono camera and stereo depth). Here is a reference in cpp which runs only yolo. Here is its equivalent in python. IF you don't need ROS, you can run these examples directly.
ok @szabi-luxonis would you mind taking a look at this, and seeing if it seems sane? I tried to implement what you suggested above.
OAK-1-MAX non stereo / non spatial YOLO example node
https://github.com/luxonis/depthai-ros/pull/262
I could use some assistance getting my Original OpenCV AI Kit: Oak-1 to work with this PR. For some reason it reports a blank "Device type:", so it can't be used with this example.
I get the following error:
[yolov4_image_node-1] what(): There is no Camera data available corresponding to the the requested cameraId
Of course due to the blank response:
[component_container-1] [INFO] [1679549436.807773594] [oak]: Device type: