Change video encoding parameters without pipeline restart

[Vavooon]

trafficstars

We expect to use the camera for UAV application, especially be able to transmit real-time video stream to ground. Since wireless link throughput may change depending on connection quality, we need to be able to decrease video bitrate when it is required, otherwise it may lead to video freezes.

So it would be great to be able to change video stream parameters without restarting entire pipeline.

Parameters we need to change:

• Bitrate (if CBR is used)
• Quality level (for VBR)
• GOP interval

[Luxonis-Brandon]

Thanks @Vavooon . This makes sense. We think it is possible to do this, and we have added this to our internal roadmap (currently item 30 on backlog).

[lucasmediaflow]

Hi guys, just wanted to say that it would be great to have this capability. If you're working on it, would you mind seeing if it would be possible to use for photos as well? I mean, for photos a video encoder is also used.

I'd like the capability to be able to trigger full res photos (THE_12_MP) and then 4K video down scaled to 1080 for crispness, without having to restart the entire pipeline.

Anyway, thanks for the amazing work you do. Cheers!

[themarpe]

Still a bit behind on this feature unfortunately. If required to be expedited please reach out.

@lucasmediaflow in your case, you may do this as of right now by using image_manip_improvements branch. Check this experiment for inspiration: https://github.com/luxonis/depthai-experiments/pull/249 Instead of cropping, you can resize the full 12MP (or cropped 4K) image coming from isp and resize it to 1080p NV12 to be able to be consumed by video encoder. Taking pictures will still be possible at 12MP.

[lucasmediaflow]

@themarpe hey mate, thank you very much for your reply. I didn't think it would be possible. I've tried pulling the imagemanip_improvements branch, but it's got an old depthai version (2.2). i'm using oak-1-poe's and it doesn't work. could it be the image_manip_refactor branch instead? I tried a basic imageManip resize from THE_4_K to 1080, with depthai version 2.13.3.0, but i'm getting this error: gist specifically this line [192.168.2.223] [65.251] [system] [critical] Fatal error. Please report to developers. Log: 'ImageManipHelper' '60' this gist has the code i'm using to initialize the camera as a class, with the pipeline and imageManip node etc... Would you be able to have a look and tell me if i'm doing something wrong? Thank you very much again. Cheers

[lucasmediaflow]

I just tried the same code as above with branch image_manip_refactor and i'm getting this error now

[lucasmediaflow]

Hi again @themarpe i've sort of got it working with image_manip_refactor branch and this very basic code:

import time import depthai as dai pipeline = dai.Pipeline() cam = pipeline.create(dai.node.ColorCamera) cam.setResolution(dai.ColorCameraProperties.SensorResolution.THE_4_K) encoder = pipeline.create(dai.node.VideoEncoder) encoderOut = pipeline.create(dai.node.XLinkOut)
encoderOut.setStreamName('1080') encoder.setDefaultProfilePreset(30, dai.VideoEncoderProperties.Profile.H264_MAIN) manip = pipeline.create(dai.node.ImageManip) manip.setMaxOutputFrameSize(3110400) manip.initialConfig.setResize(1920, 1080) manip.initialConfig.setFrameType(dai.RawImgFrame.Type.NV12) cam.isp.link(manip.inputImage) manip.out.link(encoder.input) encoder.bitstream.link(encoderOut.input)
with dai.Device(pipeline) as device: q = device.getOutputQueue(name='1080') ts = time.monotonic() with open('testy.h264', 'wb') as file: while time.monotonic() - ts < 5: frame = q.get() frame.getData().tofile(file)

the problem is that it doesn't scale the 4K frame down to a 1080 frame, but it crops the top left quadrant of the frame out to a 1080 frame it's a crop rather than a scale/resize i'm trying to achieve the same result I get with cam.setIspScale(1, 2) from a 4K frame. Is that possible at all with imageManip? If I use setIspScale() to get a 1080 frame for video, then stills from the camera seem to also be stuck at 1080 rather than 4K, does that make sense? Thanks again mate, really appreciate all you guys do and the help you provide

[themarpe]

@lucasmediaflow Good catch - it is indeed image_manip_refactor - apologies.

This behavior seems to be same as with previous ImageManip and I agree it is wrong - by default it should take the whole input and do a resize down to specified size.

I've just retested by specifying the crop manually, but also doesn't seem to go through. Let me debug this and will be back in just a moment

[lucasmediaflow]

@themarpe awesome mate, thank you very much!

[themarpe]

@lucasmediaflow found it - your above example now works as expected :)

Update the depthai using latest commit on image_manip_refactor by running the examples/install_requirements.py

[lucasmediaflow]

@themarpe it works indeed! This is awesome, thank you very much mate!!! You guys are the best!

[lucasmediaflow]

@themarpe hey mate, I've got bad news.

After 1 successful test with the code above, I went back to my main program to try to implement this, but it would crash with a segmentation fault error. I'm running it inside a docker container on a raspi.

So I went and grabbed the newest container I could find (luxonis/depthai-library:armv7-enable_multiarch_build). I installed the image_manip_refactor commit with python3 -m pip install git+https://github.com/luxonis/depthai-python.git@92788aa52a091a097b719621ae293ce55d59a82f and deleted the /depthai-python folder in the container (otherwise python would still import version 2.13.3.0).

Then I tried running only the simple test code above, and I started getting the same segmentation fault crash as with my main program.

Here is a gist with the debug output.

This is my docker run command: docker run --entrypoint /bin/bash --log-driver=journald --log-opt tag="{{.Name}}" --name maniptest -d -ti --net=host -e TZ=Pacific/Auckland --privileged -v /dev/bus/usb:/dev/bus/usb -v /home/pi/:/home/pi/ luxonis/depthai-library:armv7-enable_multiarch_build

Am I doing something wrong? Could you please have a look when you get a chance?

Thank you very much again mate

[themarpe]

@lucasmediaflow sorry for the delay.

Does the error only happen in docker? What about outside of docker? Does it happen always or do you get frames out of it occasionally as well? I think I saw occasionally a device reset without any frames being grabbed even on host. In this case, it might be that after device resets (probably FW issue) that the cleanup somewhy causes a segfault.

[lucasmediaflow]

@themarpe hey mate, thanks for getting back. I just tried it and it seems to be working fine outside of docker, both that test script above and my main program. In docker it was happening all of the time. What do you reckon it could be? I'm not going to be able to use it into production if I can't make it work in docker unfortunately. It's so close!

[themarpe]

@lucasmediaflow trying to reproduce over at my end - are you running this on Buster or Bullseye RPi OS?

[lucasmediaflow]

@themarpe this is on Buster mate, I haven't upgraded to Bullseye yet

[themarpe]

Great - then I hope I'll have a repro on my end soon. Got some ideas of what might be the cause - will let you know soon

[themarpe]

@lucasmediaflow I've pushed updated image_manip_refactor branch - can you give that a try? The wheels are building, so you might be able to skip local compilation, but either way should be fine. depthai-python commit: 0f65424c3deb66ef8f334d6723648c99f46536c7.

Most likely will address the segfault, but FW not recording but closing down is still odd - I'll take a look at it tomorrow more

[lucasmediaflow]

@themarpe hey mate, thank you very much. that newest commit seems to be doing the trick. The simple script above works just fine, same as my main program with this implemented changes, all inside docker. I couldn't make it work with luxonis/depthai-library:armv7-enable_multiarch_build container, but I reverted to luxonis/depthai-library:multiarch as a base image and it seems to be ok. Thank you very much again mate, happy christmas!

[themarpe]

Great to hear! Hopefully I'll find time sometime next week / soon after NY to merge this in with fixes as well:) Regarding luxonis/depthai-library:armv7-enable_multiarch_build docker - did you observe issues with importing numpy and other Python libraries? If so, its something I saw as well and we have a fix incoming as well, so that'll be fixed as well:)

Happy Christmas!

[lucasmediaflow]

@themarpe thanks for getting back. Yeah, I can't remember what exact issues now, but yeah, importing different libraries. opencv was one of them I think. I've run into another error a few times now, after having the cameras running for a while: [192.168.2.134] [12360.828] [system] [critical] Fatal error. Please report to developers. Log: 'OsDrvCpr' '1087' Exception in thread 192.168.2.134: Traceback (most recent call last): File "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner self.run() File "/usr/local/lib/python3.7/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/home/pi/utils.py", line 282, in startOak imgFrame = self.videoQ.tryGet() RuntimeError: Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: '44-a9-2c-33-1b-a4' (X_LINK_ERROR)' This is a script inside that docker container, that runs 2 oak-1-poe cameras. I couldn't log it with DEPTHAI_LEVEL=debug yet, but i'm trying. It just seems to happen randomly, with the cameras on idle, after an undetermined amount of time. The first time happened after 6 hours or so, the second time, within the first hour, etc... Any ideas what it might be? I think for the time being i'm going to revert back to version 2.13.3.0 to see if it fixes it. I need to deploy something, so it'll have to be a 1080 photo for now.

[themarpe]

Nothing immediately obvious comes to mind. Let me know how 2.13.3.0 does in this regard Eitherway, we'll take a look at this soon - thanks for reporting

[themarpe]

@lucasmediaflow we've pinpointed a potential cause for this issue and are working on resolving it

[lucasmediaflow]

@themarpe awesome mate, thank you very much. sorry for being the bearer of bad news :) btw, 2.13.3.0 seems to be ok so far, forgot to mention...

[lucasmediaflow]

@themarpe hey mate, I was just wondering how did you go around this issue. i'm really keen to try it when it's ready thanks again mate, happy new year!

[lucasmediaflow]

@themarpe hey mate, I was just trying the new docker v2.15.0.0-armv7 to see if the image manip issue would work. I'm getting this error: [system] [critical] Fatal error. Please report to developers. Log: 'ImageManipHelper' '61' and I'd thought I'd let you know. Here is a gist with the full debug output: https://gist.github.com/lucasmediaflow/93d86faf7fa2b4f19390ebe3e2b99f98 Cheers mate!

[themarpe]

Hi @lucasmediaflow Sorry for late answer on this - 2.15 still doesn't include the refactored ImageManip unfortunatelly. We are working on bringing it to mainline though.

If you'd like I can bring the image_manip_refactor branch up to date?

[lucasmediaflow]

@themarpe hey mate, yes please! that would be awesome if you could! Thank you very much! I can run on that branch until the changes are incorporated into the main branch, no worries... Thank you very much again mate! Cheers!

[OanaMariaVatavu]

@themarpe hey mate, yes please! that would be awesome if you could! Thank you very much! I can run on that branch until the changes are incorporated into the main branch, no worries... Thank you very much again mate! Cheers!

@themarpe, @lucasmediaflow, the image_manip_refactor branch is now up to date with the latest develop.

[lucasmediaflow]

@OanaMariaVatavu @themarpe that is awesome you guys, thank you very much!!! I'll give it a try asap... Cheers!