Could not subscribe to 2 video streams in parallel.

[kkishore9891]

I am using a slightly modified version of the LiveviewSample code for our use case. But we have just changed the names of the function and the camera positions. The issue that we are facing is that we can either subscribe to the thermal camera feed or the wide camera feed at the time, as shown by commenting out liveview_camera->StartWideCameraStream or liveview_camera->StartThermalCameraStream. auto *liveview_camera = new LiveviewSample();

    liveview_camera->StartWideCameraStream(&WideImageCallback, &wide_camera_data_);
    RCLCPP_INFO(rclcpp::get_logger(logger_name), "Selected Wide Camera Stream");

    // liveview_camera->StartThermalCameraStream(&ThermalImageCallback, &thermal_camera_data_);
    // RCLCPP_INFO(rclcpp::get_logger(logger_name), "Selected Thermal Camera Stream");
    

But when we uncomment both the lines to subscribe to both video streams in parallel, none of the callbacks receive any images. The PSDK is still functioning well as I can receive widget data in widget callbacks and so on. Is this a shortcoming of PSDK or does subscribing from two image streams at the same time require special procedures?

[dji-dev]

Agent comment from Leon in Zendesk ticket #102279:

Dear Developer,

Hello, thank you for contacting DJI.

May I ask which model are you using? This problem may be related to the model of the machine.

Thank you for your support of DJI products.

Best Regards, DJI Dajiang innovation SDK technical support

°°°

[kkishore9891]

We are using the DJI Matrice M30T model.

[dji-dev]

Agent comment from Leon in Zendesk ticket #102279:

Dear Developer,

Hello, thank you for contacting DJI.

The M30T model supports subscribing to two camera video streams at the same time. If you need to customize the streams of two lenses at the same time, you can refer to this method: That is, call the DjiLiveview_StartH264Stream function to specify two different video sources.

Thank you for your support of DJI products.

Best Regards, DJI Dajiang innovation SDK technical support
image.png

°°°

[kkishore9891]

We are currently using the following codes.

In our dji_psdk_ros2_node.cpp we have:

    //CAMERA SETUP
    try
    {
        auto *liveview_camera = new LiveviewSample();

        liveview_camera->StartWideCameraStream(&WideImageCallback, &wide_camera_data_);
        RCLCPP_INFO(rclcpp::get_logger(logger_name), "Selected Wide Camera Stream");

        liveview_camera->StartThermalCameraStream(&ThermalImageCallback, &thermal_camera_data_);
        RCLCPP_INFO(rclcpp::get_logger(logger_name), "Selected Thermal Camera Stream");

        // camera_stream_handler(camera_mount_position_, &data_);
    }
    catch (const std::exception &e)
    {
        RCLCPP_INFO(rclcpp::get_logger(logger_name), "Error while camera stream handler: %s", e.what());
    }
    catch (...)
    {
        RCLCPP_INFO(rclcpp::get_logger(logger_name), "Unknown error while camera stream handler");
    }

In our modified test_liveview.cpp we have:

T_DjiReturnCode LiveviewSample::StartWideCameraStream(CameraImageCallback callback, void *userData)
{
    auto deocder = streamDecoder.find(DJI_LIVEVIEW_CAMERA_POSITION_NO_1);
    // USER_LOG_ERROR("Starting wide");
    if ((deocder != streamDecoder.end()) && deocder->second)
    {
        deocder->second->init();
        deocder->second->registerCallback(callback, userData);
        USER_LOG_ERROR("Starting Wide Camera Streaming");
        return DjiLiveview_StartH264Stream(DJI_LIVEVIEW_CAMERA_POSITION_NO_1, DJI_LIVEVIEW_CAMERA_SOURCE_M30T_WIDE,
                                           LiveviewConvertH264ToRgbCallback);
    }
    else
    {
        return DJI_ERROR_SYSTEM_MODULE_CODE_NOT_FOUND;
    }
}

T_DjiReturnCode LiveviewSample::StartThermalCameraStream(CameraImageCallback callback, void *userData)
{
    auto deocder = streamDecoder.find(DJI_LIVEVIEW_CAMERA_POSITION_NO_1);

    if ((deocder != streamDecoder.end()) && deocder->second)
    {
        deocder->second->init();
        deocder->second->registerCallback(callback, userData);
        USER_LOG_ERROR("Starting Thermal Camera Streaming");
        return DjiLiveview_StartH264Stream(DJI_LIVEVIEW_CAMERA_POSITION_NO_1, DJI_LIVEVIEW_CAMERA_SOURCE_M30T_IR,
                                           LiveviewConvertH264ToRgbCallback);
    }
    else
    {
        return DJI_ERROR_SYSTEM_MODULE_CODE_NOT_FOUND;
    }
}

But the callbacks are not receiving any images unless we comment out one of the liveview_camera->StartCameraStream()

functions.

[dji-dev]

Agent comment from Leon in Zendesk ticket #102279:

Dear Developer,

Hello, thank you for contacting DJI.

Thank you for your patience. Got it, we will try to reproduce your problem in a local C++ routine and will contact you again soon.

Thank you for your support of DJI products.

Best Regards, DJI Dajiang innovation SDK technical support

°°°

[kkishore9891]

Hello! Are there any updates regarding this. Having two video streams would be of great help to train AI models that work on both RGB and thermal modalities.

[dji-dev]

Agent comment from Leon in Zendesk ticket #102279:

Dear Developer,

Hello, thank you for contacting DJI.

Thank you for your patience. At present, we can verify the problem of being unable to obtain the two videos and have synchronized them with the R&D team for analysis and confirmation. If there is any progress, we will synchronize them with you again.

Thank you for your support of DJI products.

Best Regards, DJI Dajiang innovation SDK technical support

°°°

[kkishore9891]

Hello! I would like to know if there are any updates with regards to this issue. Thanks.

[mstampa]

Hello, I would also very much appreciate an update on this. Can the fix be expected anytime soon?

I'm preparing a series of experiments which need to be completed before October. They involve the wide-angle and thermal camera, as well as the pose information, of an M30T. I was quite disappointed to learn that getting the streams simultaneously is not possible at the moment.

As a disatisfactory but workable backup plan, I could take a photo series with the regular DJI Pilot app and extract the pose information from the metadata. For this option, it'd be really helpful if a) The interval between photos could be decreased from 2s to 1s or even lower. b) The precision of the drone and gimbal angles in the metadata could be increased. They currently have only one lonely decimal place.

Thanks!