OAK-D-PRO-POE shutdown after a while

[khuechuong]

ROS Noetic OAK-D-PRO-POE

So the oak-d-pro-poe was fine when it starts, but for some reason, after a while it gives me this

[1844301041E8E9F400] [169.254.1.221] [1723044660.375] [host] [warning] Monitor thread (device: 1844301041E8E9F400 [169.254.1.221]) - ping was missed, closing the device connection
[ERROR]ros.depthai_ros_driver][produceDiagnostics:80]: No data on logger queue!
[ERROR]ros.depthai_ros_driver][diagCB:61]: Camera diagnostics error: Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: 'sys_logger_queue' (X_LINK_ERROR)'
[ERROR]ros.depthai_ros_driver][produceDiagnostics:80]: No data on logger queue!
[ERROR]ros.depthai_ros_driver][diagCB:61]: Camera diagnostics error: Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: 'sys_logger_queue' (X_LINK_ERROR)'

and then shutdown.

[Serafadam]

Hi, this might be due to high CPU usage on the camera, you can monitor it either via diagnostics topic or by enabling debug information - export DEPTHAI_DEBUG=1, you can also set a parameter camera_i_restart_on_diagnostics_error: true to have the camera restart after encountering an error

[naman-gupta99]

Hi, this might be due to high CPU usage on the camera, you can monitor it either via diagnostics topic or by enabling debug information - export DEPTHAI_DEBUG=1, you can also set a parameter camera_i_restart_on_diagnostics_error: true to have the camera restart after encountering an error

Hey! What should I look for in the diagnostics topic or the debug logs to find that there is high CPU usage? And even if that is the issue is there no other solution than allowing the camera to keep restarting?

[naman-gupta99]

@khuechuong Were you able to figure this issue out? @Serafadam Can you please help?

[Serafadam]

Hi, sorry, I was out sick last week. You can check the CPU usage of the camera either by looking on diagnostics topic or by enabling debug logs via running export DEPTHAI_DEBUG=1 before running the driver. In case of PoE camera you might also try updating the bootloader of the camera (more information here)

[naman-gupta99]

@Serafadam I did add these but I couldn't find anything related to CPU Usage.

header:
  stamp:
    sec: 1740806521
    nanosec: 556034201
  frame_id: ''
status:
- level: "\x02"
  name: 'rtabmap: Sync status'
  message: No events recorded.; No data since last update.
  hardware_id: none
  values:
  - key: Events in window
    value: '0'
  - key: Events since startup
    value: '0'
  - key: Duration of window (s)
    value: '4.999979'
  - key: Actual frequency (Hz)
    value: '0.000000'
  - key: Target frequency (Hz)
    value: '0.000000'
  - key: Maximum acceptable frequency (Hz)
    value: '0.000000'
  - key: 'Earliest timestamp delay:'
    value: '0.000000'
  - key: 'Latest timestamp delay:'
    value: '0.000000'
  - key: 'Earliest acceptable timestamp delay:'
    value: '-1.000000'
  - key: 'Latest acceptable timestamp delay:'
    value: '5.000000'
  - key: 'Late diagnostic update count:'
    value: '0'
  - key: 'Early diagnostic update count:'
    value: '0'
  - key: 'Zero seen diagnostic update count:'
    value: '0'
---
header:
  stamp:
    sec: 1740806521
    nanosec: 643101486
  frame_id: ''
status:
- level: "\x02"
  name: 'rtabmap_viz: Sync status'
  message: No events recorded.; No data since last update.
  hardware_id: none
  values:
  - key: Events in window
    value: '0'
  - key: Events since startup
    value: '0'
  - key: Duration of window (s)
    value: '2.999869'
  - key: Actual frequency (Hz)
    value: '0.000000'
  - key: Target frequency (Hz)
    value: '0.000000'
  - key: Maximum acceptable frequency (Hz)
    value: '0.000000'
  - key: 'Earliest timestamp delay:'
    value: '0.000000'
  - key: 'Latest timestamp delay:'
    value: '0.000000'
  - key: 'Earliest acceptable timestamp delay:'
    value: '-1.000000'
  - key: 'Latest acceptable timestamp delay:'
    value: '5.000000'
  - key: 'Late diagnostic update count:'
    value: '0'
  - key: 'Early diagnostic update count:'
    value: '0'
  - key: 'Zero seen diagnostic update count:'
    value: '0'
---
header:
  stamp:
    sec: 1740806522
    nanosec: 52955374
  frame_id: ''
status:
- level: "\x02"
  name: 'oak: sys_logger'
  message: 'Communication exception - possible device error/misconfiguration. Original message ''Couldn''t read data from stream: ''sys_logger_...'
  hardware_id: oak_14442C103141E0D600_OAK-D-LITE
  values: []
---
header:
  stamp:
    sec: 1740806522
    nanosec: 556022533
  frame_id: ''
status:
- level: "\x02"
  name: 'rtabmap: Sync status'
  message: No events recorded.; No data since last update.
  hardware_id: none
  values:
  - key: Events in window
    value: '0'
  - key: Events since startup
    value: '0'
  - key: Duration of window (s)
    value: '4.999971'
  - key: Actual frequency (Hz)
    value: '0.000000'
  - key: Target frequency (Hz)
    value: '0.000000'
  - key: Maximum acceptable frequency (Hz)
    value: '0.000000'
  - key: 'Earliest timestamp delay:'
    value: '0.000000'
  - key: 'Latest timestamp delay:'
    value: '0.000000'
  - key: 'Earliest acceptable timestamp delay:'
    value: '-1.000000'
  - key: 'Latest acceptable timestamp delay:'
    value: '5.000000'
  - key: 'Late diagnostic update count:'
    value: '0'
  - key: 'Early diagnostic update count:'
    value: '0'
  - key: 'Zero seen diagnostic update count:'
    value: '0'
---
header:
  stamp:
    sec: 1740806522
    nanosec: 643193338
  frame_id: ''
status:
- level: "\x02"
  name: 'rtabmap_viz: Sync status'
  message: No events recorded.; No data since last update.
  hardware_id: none
  values:
  - key: Events in window
    value: '0'
  - key: Events since startup
    value: '0'
  - key: Duration of window (s)
    value: '3.999932'
  - key: Actual frequency (Hz)
    value: '0.000000'
  - key: Target frequency (Hz)
    value: '0.000000'
  - key: Maximum acceptable frequency (Hz)
    value: '0.000000'
  - key: 'Earliest timestamp delay:'
    value: '0.000000'
  - key: 'Latest timestamp delay:'
    value: '0.000000'
  - key: 'Earliest acceptable timestamp delay:'
    value: '-1.000000'
  - key: 'Latest acceptable timestamp delay:'
    value: '5.000000'
  - key: 'Late diagnostic update count:'
    value: '0'
  - key: 'Early diagnostic update count:'
    value: '0'
  - key: 'Zero seen diagnostic update count:'
    value: '0'
---
header:
  stamp:
    sec: 1740806523
    nanosec: 52457378
  frame_id: ''
status:
- level: "\x02"
  name: 'oak: sys_logger'
  message: 'Communication exception - possible device error/misconfiguration. Original message ''Couldn''t read data from stream: ''sys_logger_...'
  hardware_id: oak_14442C103141E0D600_OAK-D-LITE
  values: []
---
header:
  stamp:
    sec: 1740806523
    nanosec: 556030103
  frame_id: ''
status:
- level: "\x02"
  name: 'rtabmap: Sync status'
  message: No events recorded.; No data since last update.
  hardware_id: none
  values:
  - key: Events in window
    value: '0'
  - key: Events since startup
    value: '0'
  - key: Duration of window (s)
    value: '5.000002'
  - key: Actual frequency (Hz)
    value: '0.000000'
  - key: Target frequency (Hz)
    value: '0.000000'
  - key: Maximum acceptable frequency (Hz)
    value: '0.000000'
  - key: 'Earliest timestamp delay:'
    value: '0.000000'
  - key: 'Latest timestamp delay:'
    value: '0.000000'
  - key: 'Earliest acceptable timestamp delay:'
    value: '-1.000000'
  - key: 'Latest acceptable timestamp delay:'
    value: '5.000000'
  - key: 'Late diagnostic update count:'
    value: '0'
  - key: 'Early diagnostic update count:'
    value: '0'
  - key: 'Zero seen diagnostic update count:'
    value: '0'
---
header:
  stamp:
    sec: 1740806523
    nanosec: 643059316
  frame_id: ''
status:
- level: "\x02"
  name: 'rtabmap_viz: Sync status'
  message: No events recorded.; No data since last update.
  hardware_id: none
  values:
  - key: Events in window
    value: '0'
  - key: Events since startup
    value: '0'
  - key: Duration of window (s)
    value: '4.998765'
  - key: Actual frequency (Hz)
    value: '0.000000'
  - key: Target frequency (Hz)
    value: '0.000000'
  - key: Maximum acceptable frequency (Hz)
    value: '0.000000'
  - key: 'Earliest timestamp delay:'
    value: '0.000000'
  - key: 'Latest timestamp delay:'
    value: '0.000000'
  - key: 'Earliest acceptable timestamp delay:'
    value: '-1.000000'
  - key: 'Latest acceptable timestamp delay:'
    value: '5.000000'
  - key: 'Late diagnostic update count:'
    value: '0'
  - key: 'Early diagnostic update count:'
    value: '0'
  - key: 'Zero seen diagnostic update count:'
    value: '0'
---
header:
  stamp:
    sec: 1740806524
    nanosec: 52341937
  frame_id: ''
status:
- level: "\x02"
  name: 'oak: sys_logger'
  message: 'Communication exception - possible device error/misconfiguration. Original message ''Couldn''t read data from stream: ''sys_logger_...'
  hardware_id: oak_14442C103141E0D600_OAK-D-LITE
  values: []
---
header:
  stamp:
    sec: 1740806524
    nanosec: 555944614
  frame_id: ''
status:
- level: "\x02"
  name: 'rtabmap: Sync status'
  message: No events recorded.; No data since last update.
  hardware_id: none
  values:
  - key: Events in window
    value: '0'
  - key: Events since startup
    value: '0'
  - key: Duration of window (s)
    value: '4.999944'
  - key: Actual frequency (Hz)
    value: '0.000000'
  - key: Target frequency (Hz)
    value: '0.000000'
  - key: Maximum acceptable frequency (Hz)
    value: '0.000000'
  - key: 'Earliest timestamp delay:'
    value: '0.000000'
  - key: 'Latest timestamp delay:'
    value: '0.000000'
  - key: 'Earliest acceptable timestamp delay:'
    value: '-1.000000'
  - key: 'Latest acceptable timestamp delay:'
    value: '5.000000'
  - key: 'Late diagnostic update count:'
    value: '0'
  - key: 'Early diagnostic update count:'
    value: '0'
  - key: 'Zero seen diagnostic update count:'
    value: '0'
---

And I have the OAK-D Lite Camera.

I am sorry if I am asking very basic questions but I am very new to this, so any help would be really appreciated.

[naman-gupta99]

@Serafadam Can you please check this and help?

[Serafadam]

Hi, it seems that the camera encounters some errors and shuts down, could you run export DEPTHAI_DEBUG=1 before running the driver and paste the logs?

[naman-gupta99]

Hello @Serafadam. Thank you for your response. You can see the log file here.

I ran Visual SLAM on my machine, and the camera worked fine until the rover was stationary. As soon as I moved the rover, it errored out and shut down.

logfile.txt

[Serafadam]

Thanks for the report, it looks like the device is running fine but it seems that there is some issue related to diagnostics which could be firmware related. Could you disable diagnostics? You can do that by setting pipeline_gen.i_enable_diagnostics: false parameter

[naman-gupta99]

Hi @Serafadam,
Apologies for the delayed response.

I made the following change to the ugv_ws/src/ugv_main/ugv_vision/config/oak_d_lite.yaml file:

/oak:
  ros__parameters:
    camera:
      i_enable_imu: false
      i_enable_ir: false
      i_nn_type: none
      i_enable_diagnostics: false
    stereo:
      i_subpixel: true

Could you please confirm if this is the correct location for these parameters? I understand this setup may differ from yours since it's running on a different platform, so no worries if it's outside the scope of what you can answer.

That said, even with these changes, the camera still turns off after a short time.

[naman-gupta99]

@Serafadam Can you please check this?

[jagruth-medavarapu]

Hi @naman-gupta99, were you able to solve this? I am facing the same error but with different camera.

[naman-gupta99]

Yeah. For me it was a power issue. I connected a seperate direct power source for the camera, instead through the Ras Pi. That fixed the issue.