Issues using the VideoPreviewerSdkAdapter and DJIVideoPreviewer

[alexsullivan114]

We're attempting to use the the DJI iOS SDK and we're having some issues getting first person video to reliably work.

We've found that if you start the product connection process too early in the apps lifecycle (i.e. immediately after app launch or within a couple seconds of app launch) video simply will not show. If you background and reopen the app there will often be severe artifacts in the video stream.

However, if we wait 5-10 seconds after app launch to start the product registration and connection process things mostly work fine (though we still occasionally see artifacts in the video after backgrounding the app and coming back again).

When we get the artifacts, we typically see a stream of errors that look like the following:

[h264 @ 0x118b85c10] concealing 2780 DC, 2780 AC, 2780 MV errors in P frame
[h264 @ 0x1188f2410] error while decoding MB 69 10, bytestream -12
[h264 @ 0x1188f2410] Cannot use next picture in error concealment

Which appears to be an FFMPeg issue.

When we see no video, we'll typically see 10-20 logs that look like this:

[h264 @ 0x10887c210] non-existing PPS 0 referenced

Which also appears to be an FFMPeg issue.

Right now, we have a custom UIView that's supposed to house the drone video that's quite simple:

class DroneVideoView: UIView {
  override init(frame: CGRect) {
    super.init(frame: frame)
    startStreaming()
  }

  required init?(coder aDecoder: NSCoder) {
    fatalError("Unsupported init with coder")
  }

  @objc private func stopStreaming() {
    cameraManager.stopStreamingDroneVideo()
  }

  @objc private func startStreaming() {
    cameraManager.streamDroneVideo(to: self)
  }

  deinit {
    stopStreaming()
  }
}

It delegates most of the work to a CameraManager class.

The startStreaming method on the CameraManager waits for the DJI product to be connected and then calls the following:

DJIVideoPreviewer.instance()?.setView(view)
self?.previewerAdapter = VideoPreviewerSDKAdapter.withDefaultSettings()
self?.previewerAdapter?.start()
self?.previewerAdapter?.setupFrameControlHandler()
DJIVideoPreviewer.instance()?.start()

The stopStreaming method looks similar:

self.previewerAdapter?.stop()
DJIVideoPreviewer.instance()?.close()
DJIVideoPreviewer.instance()?.unSetView()

Are we missing anything? We're not using the UI SDK at the moment because of issues around the use_framework! directive in cocoapods.

EDIT:

After a lot of sleuthing and logging, it looks like the video is failing to render because of an error when setting up the DJIMovieGLView within DJIVideoPreviewer. Specifically, in the initWithFrame method of DJIMovieGLView the GLenum status of GL_FRAMEBUFFER returns GL_FRAMEBUFFER_INCOMPLETE_ATTACHMENT, causing the DJIMovieGLView to initialize to nil. I'm not sure why this sometimes happens and sometimes doesn't but that appears to be the underlying cause. I'm not well versed enough in OpenGL to debug much further unfortunately.

EDIT: It looks like unsetting the view and resetting the view fixes the issue. I'm going to leave this up for now because that solution is a hack, but it does provide a path forward.

[dji-dev]

Public comment from Lisa Fedane in Zendesk ticket #27960:

Hi Alex - are you seeing this if you run our iOS Sample App, as is?

Lisa Fedane DJI Developer Support

[alexsullivan114]

No, but the sample app also doesn't immediately show the video so I wouldn't expect to see the error.

[yulinxfx]

I used the SDK to broadcast live, but the live broadcast was delayed a lot and kept piling up, which made it impossible to use. The SDK was too bad

[brien-crean]

@alexsullivan114 Did you ever find a solution or do you have some sample code on unsetting and resetting the view? I am seeing similar issues in the latest version of the SDK

[h264 @ 0x13daf9010] non-existing PPS 63 referenced

I tried the following but the issue remains

[DJIVideoPreviewer.instance unSetView];
[DJIVideoPreviewer.instance reset];
[DJIVideoPreviewer.instance setView:videoFeedView];
[DJIVideoPreviewer.instance start];

I did notice that changing the photoAspectRatio of the DJICamera class seems to reset the view and I see the FPV output again. I also see this in the console: app[18266:5316924] [vs]stream info changed : <NSThread: 0x281ff5600>{number = 24, name = com.dji.videoPreviewer.decodeThread}

But setting the aspect ratio seems like a worse hack than simply resetting the view

[sgallo0692]

@brien-crean did you ever find a solution to your issue? Or - does your implementation still actually work with some performance issues? I'm trying to use the streaming sdk adapter and followed their objc example and can't get it to work. saw you had a similar issue and was hoping you had a functioning example of getting the video feed

[thealch3m1st]

I was trying the FPV demo and I was getting the same error for my Mavic air 2 this is what fixed it: -(void)videoFeed:(DJIVideoFeed *)videoFeed didUpdateVideoData:(NSData *)videoData { [videoFeed parseDecodingAssistInfoWithBuffer:videoData.bytes length:videoData.length assistInfo:&self->_assist]; [[DJIVideoPreviewer instance] push:(uint8_t *)videoData.bytes length:(int)videoData.length]; }