VideoFeedUpdate hook not working

[jeryini]

Hello!

I'm trying to use hooks on FPVWidgetState, specifically data class VideoFeedUpdate(val videoBuffer: ByteArray?, val size: Int) : FPVWidgetState() to get video frame each time it's updated. I'm using the following example in Java to subscribe to updates:

fpvWidget.getWidgetStateUpdate()
            .observeOn(AndroidSchedulers.mainThread())
            .subscribe(fpvWidgetState -> {
                if (fpvWidgetState instanceof FPVWidget.FPVWidgetState.ProductConnected) {
                    FPVWidget.FPVWidgetState.ProductConnected productConnected = (FPVWidget.FPVWidgetState.ProductConnected) fpvWidgetState;
                    Log.d(TAG, "Product connected: " + productConnected.isConnected());
                }
                if (fpvWidgetState instanceof FPVWidget.FPVWidgetState.VideoFeedUpdate) {
                    FPVWidget.FPVWidgetState.VideoFeedUpdate videoFeedUpdate = (FPVWidget.FPVWidgetState.VideoFeedUpdate) fpvWidgetState;
                    Log.d(TAG, "Feed size: " + videoFeedUpdate.getSize());
                    Log.d(TAG, "Buffer: " + Arrays.toString(videoFeedUpdate.getVideoBuffer()));
                }
            });

Unfortunately I don't see anything in the logs regarding outputted feed size and buffer. I've also tried ProductConnected state and that actually works fine, I can see the output in the log. Do I have to set up something else on the codecmanager or FPVWidget level to get the back the update for the video feed?

[dji-lisafedane]

@jeryini - thanks for reporting on this bug and the good news is that there'll be a fix for this in the next beta release

[jeryini]

Hello @dji-lisafedane !

Nice to hear that! Can't wait for it.

[alainpimentel]

@jeryini btw thanks for trying and reporting this!

[jeryini]

Hello @alainpimentel !

Do you maybe have any ETA when this will be fixed. Many thanks!

Cheers, Jernej

[alainpimentel]

Hi @jeryini! We have a fix that will come out with the new MSDK 4.14. Keep an eye on https://developer.dji.com/user/roadmap/#msdk for more updates on the release date. Our next beta release will be shortly after 4.14 is out.

[jeryini]

Hello @alainpimentel @siddutgikar !

First congratulations on the new awesome release with so many new features. Though I'm still having issues with getting VideoFeedUpdated model state updates.

My configuration:

• Android Mobile SDK 4.14-trial1
• Mobile-UXSDK-Beta-Android v0.5.0
• DJI Mavic 2 Zoom, firmware version: 01.00.0670

I'm using the following code:

fpvWidget.getWidgetStateUpdate()
                        .observeOn(AndroidSchedulers.mainThread())
                        .subscribe(fpvWidgetState -> {
                            Log.d(TAG, fpvWidgetState.toString());
                            if (fpvWidgetState instanceof FPVWidget.ModelState.VideoFeedUpdated) {
                                FPVWidget.ModelState.VideoFeedUpdated videoFeedUpdated = (FPVWidget.ModelState.VideoFeedUpdated) fpvWidgetState;
                                Log.d(TAG, "Feed size: " + videoFeedUpdated.getSize());
                                Log.d(TAG, "Buffer: " + Arrays.toString(videoFeedUpdated.getVideoBuffer()));
                            }
                        });

My logs show the following:

2020-12-22 13:19:16.039 26881-26881/co.dronecontrol.app D/GroundControlStationActivity: VideoFeedSourceUpdated(videoFeed=UNKNOWN)
2020-12-22 13:19:16.039 26881-26881/co.dronecontrol.app D/GroundControlStationActivity: VideoFeedSourceUpdated(videoFeed=CAMERA)
2020-12-22 13:19:16.039 26881-26881/co.dronecontrol.app D/GroundControlStationActivity: OrientationUpdated(orientation=UNKNOWN)
2020-12-22 13:19:16.039 26881-26881/co.dronecontrol.app D/GroundControlStationActivity: CameraNameUpdated(cameraName=)
2020-12-22 13:19:16.040 26881-26881/co.dronecontrol.app I/chatty: uid=10376(co.dronecontrol.app) identical 2 lines
2020-12-22 13:19:16.040 26881-26881/co.dronecontrol.app D/GroundControlStationActivity: CameraNameUpdated(cameraName=)
2020-12-22 13:19:16.040 26881-26881/co.dronecontrol.app D/GroundControlStationActivity: CameraNameUpdated(cameraName=Mavic 2 Zoom Camera)
2020-12-22 13:19:16.040 26881-26881/co.dronecontrol.app I/chatty: uid=10376(co.dronecontrol.app) identical 1 line
2020-12-22 13:19:16.040 26881-26881/co.dronecontrol.app D/GroundControlStationActivity: CameraNameUpdated(cameraName=Mavic 2 Zoom Camera)
2020-12-22 13:19:16.040 26881-26881/co.dronecontrol.app D/GroundControlStationActivity: CameraSideUpdated(cameraSide=Unknown)
2020-12-22 13:19:16.040 26881-26881/co.dronecontrol.app I/chatty: uid=10376(co.dronecontrol.app) identical 5 lines
2020-12-22 13:19:16.040 26881-26881/co.dronecontrol.app D/GroundControlStationActivity: CameraSideUpdated(cameraSide=Unknown)
2020-12-22 13:19:16.040 26881-26881/co.dronecontrol.app D/GroundControlStationActivity: ProductConnected(isConnected=false)
2020-12-22 13:19:16.040 26881-26881/co.dronecontrol.app D/GroundControlStationActivity: ProductConnected(isConnected=true)

As you can see almost all model state events do show up, except for the VideoFeedUpdated state. Is there perhaps some additional configuration needed?

[alainpimentel]

@jeryini great to see you are already testing the new release! Good find. Something is still broken so we'll be taking a look.

[jeryini]

Hello @alainpimentel !

Have you maybe pinpointed the issue? Is this something that can be fixed in this library, or does it need another update at the MSDK level?

[snmillar]

Hello @jeryini

I've been looking into the issue and I found that the raw video feed is not available for Mavic 2 Zoom because it needs lens distortion calibration, so the transcoded video feed should be used instead.

Would the data from the transcoded video feed work for your purposes? I can add another hook that gets this data.

Or if you would like to get the transcoded video feed yourself without waiting for an update to this library, you can use the following method:

VideoFeeder.getInstance().provideTranscodedVideoFeed().addVideoDataListener(VideoDataListener listener)

[jeryini]

Hello @snmillar !

This might work. Currently I'm using YuvData callback:

codecManager.enabledYuvData(true);
codecManager.setYuvDataCallback

I'm curious, what is actually the difference between subscribing to video via this transcoded video feed vs. using YUV data callback? Is there a difference in video quality? Also I searched inside this library and noticed you don't use YUV data callback to get the video feed, right?

In the end I want to use subscribe to your video feed mechanism because I'm currently using my own version to parse the generated YUV data inside onYuvDataReceived method to get the raw video feed. And that implementation is working fine, but I think it has a bit worse performance then what I see in the official DJI GO app.

[alainpimentel]

@jeryini we are looking into the difference "between subscribing to video via this transcoded video feed vs. using YUV data callback". Will keep you posted when we have a good answer. In the meantime, hope the MSDK API helps you move forward. We will be adding this as a hook in the next release.

[hummelslei]

As I know, the onYuvDataReceived will give the YUV format data and the provideTranscodedVideoFeed provide the H264 data. Also, the provideTranscodedVideoFeed channel has method setTranscodingDataRate to change the rate.

[jeryini]

Hello @alainpimentel !

Actually having YUV data available via subscription would be nice! Though based on my experience, if you enable YUV data, for some reason main FPV TextureView stops displaying camera image. Not sure if you can get around this.

[alainpimentel]

We will need to explore why enabling YUV data causes issues tho, do you see that with other drones other than Mavic 2 Zoom? If you have any available.

[jeryini]

Yes I tried with Mavic Mini and it's the same. If I remember correctly from one of the Github issue threads in Mobile SDK project, it's due to DJICodecManager unable to render to SurfaceView at the same time while processing YUV data. I actually tried to create two instances of DJICodecManager, enabling YUV on only one, but the result was the same.

[natxopedreira]

Hello

I want to get video frames from a fpvwidget, im trying to suscribe using getWidgetStateUpdate() but is not showing as a method of fpvwidget..... what im missing?????

Also is not clear to me if is possible or not to get that video frame as according to this thread is this hook not working? is there any workarround¿

Thanks and sorry for the newbie question

[jeryini]

Hello @alainpimentel, @snmillar !

Do you have any progress with exposing YUV data? It would help me a lot and probably also other users. I'm having a hard time getting YUV data while DJICodecManager would still be able to render video to SurfaceView, I think it's mutually exclusive, i.e. you either get raw YUV data or you render it to TextureView. I even checked and tried this example (https://github.com/DJI-Mobile-SDK-Tutorials/Android-VideoStreamDecodingSample/issues/29#issuecomment-767026313), but came across another issue. Basically I tried to use that example to pass the raw byte buffer that I get via your VideoFeedUpdate hook and then pass it to DJIVideoStreamDecoder:

DJIVideoStreamDecoder.getInstance().changeSurface(null);
DJIVideoStreamDecoder.getInstance().setYuvDataListener(djiDroneData);
fpvWidget.getWidgetStateUpdate()
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(fpvWidgetState -> {
        if (fpvWidgetState instanceof FPVWidget.ModelState.VideoFeedUpdated) {
            FPVWidget.ModelState.VideoFeedUpdated videoFeedUpdated = (FPVWidget.ModelState.VideoFeedUpdated) fpvWidgetState;
            Log.d(TAG, "Feed size: " + videoFeedUpdated.getSize());
            DJIVideoStreamDecoder.getInstance().parse(videoFeedUpdated.getVideoBuffer(), videoFeedUpdated.getSize());
        }
    });

But as mentioned in that issue comment, I just get blank screen now, no video being rendered to textureview_fpv.

[jeryini]

Hello @alainpimentel @snmillar !

I've just seen new release for Beta 5.1, great job! I've quickly scrolled over the codebase for FPVWidget.kt but don't see any option to subscribe to YUV data. I guess that feature didn't make it to the latest release? Are you maybe still planning to support it in the later versions? This would help us third party developers a lot to be able to just seamlessly subscribe to the stream like I can do now for VideoFeedUpdated.

[mayurpatil307]

Hi I am facing same issue, I'm trying to use hooks on FPVWidgetState,

fpvWidget.getWidgetStateUpdate()
            .observeOn(AndroidSchedulers.mainThread())
            .subscribe(fpvWidgetState -> {
                if (fpvWidgetState instanceof FPVWidget.FPVWidgetState.ProductConnected) {
                    FPVWidget.FPVWidgetState.ProductConnected productConnected = (FPVWidget.FPVWidgetState.ProductConnected) fpvWidgetState;
                    Log.d(TAG, "Product connected: " + productConnected.isConnected());
                }
                if (fpvWidgetState instanceof FPVWidget.FPVWidgetState.VideoFeedUpdate) {
                    FPVWidget.FPVWidgetState.VideoFeedUpdate videoFeedUpdate = (FPVWidget.FPVWidgetState.VideoFeedUpdate) fpvWidgetState;
                    Log.d(TAG, "Feed size: " + videoFeedUpdate.getSize());
                    Log.d(TAG, "Buffer: " + Arrays.toString(videoFeedUpdate.getVideoBuffer()));
                }
            });

But, the logs are not getting executed.

I tried:

 fpvWidget.setCodecManagerCallback(new FPVWidget.CodecManagerCallback() {
            @Override
            public void onCodecManagerChanged(@Nullable DJICodecManager codecManager) {
                Log.d(TAG, "Codec manager changed");

                codecManager = new DJICodecManager(activity, null, 1280, 720, UsbAccessoryService.VideoStreamSource.Camera);
                codecManager.enabledYuvData(true);

                Log.d(TAG, "YUV data enabled");

                codecManager.setYuvDataCallback(new DJICodecManager.YuvDataCallback() {
                    @Override
                    public void onYuvDataReceived(MediaFormat mediaFormat, ByteBuffer yuvFrame, int dataSize, int width, int height) {
                        Log.d(TAG, "New YUV data received");
                    }
                });
            }
        });

I observed following: The setYuvDataCallback is not called continuously while we are in FPVWidget activity. Its called continuously when we go back to previous activity or any other activity from FPVWidget's activity.

[jeryini]

Hello @mayurpatil307 !

Are you maybe using Mavic 2 (Pro/Zoom)? There's an issue with those drones, you need to enable some other stuff via SDK calls, see above posts for solution.