Support offset of image capturing in multiple modules for interweaving

[mationai]

If modules can synced so image capturing can be offset, images can be interweaved to enable FPS of 240 or more. This will bring in a new segment of customers who had been buying expensive systems for high speed systems.

[Luxonis-Brandon]

Thanks @fuzzthink . Depending on the number of cameras needed, we may be able to do some interesting things here.

1. How many cameras are you needing? The stereo cameras on DepthAI could perhaps be initialized exactly out of phase for example.
2. So is the plan to simply record the frames as RAW output? I ask because I'm not sure if the encoder can keep up with this framerate.

[Luxonis-Brandon]

This will likely be supported by https://github.com/luxonis/depthai/issues/214

[mationai]

Thank you.

1. Not sure yet, will probably start with two. Ability to initialize out of phase will be great!

2. Yes, can of load encoding if encoder isn't able to keep up.

[Luxonis-Brandon]

Hi @fuzzthink ,

Thanks! And what resolution is needed at this rate. If we re-write our driver we can theoretically get 210fps at 640x400 from the OV9282 directly.

Thoughts?

Thanks, Brandon

[mationai]

May need a bit higher than that. Will greyscale bump the resolution or fps?

[Luxonis-Brandon]

So the theoritical limits (require driver re-write) for the OV9282 are: 1280x800 - 120fps 1280x720 - 130fps 640x480 - 180fps 640x400 - 210fps

There are other faster sensors out there that could do for example over 300fps at 1280x800 (or higher) but we'd have to implement the driver for these.

Thoughts?

[mationai]

line 2 should be 130fps?

[Luxonis-Brandon]

Yes, edited thanks!

[mationai]

Once these options are enabled, users will be able to configure them, right?

[Luxonis-Brandon]

Yes, and the current settings are viewable by running ./depthai_demo.py -h, and here are the pertinent results for resolution and framerate:

  -rgbr {1080,2160,3040}, --rgb_resolution {1080,2160,3040}
                        RGB cam res height: (1920x)1080, (3840x)2160 or
                        (4056x)3040. Default: 1080
  -rgbf RGB_FPS, --rgb_fps RGB_FPS
                        RGB cam fps: max 118.0 for H:1080, max 42.0 for
                        H:2160. Default: 30.0
  -cs COLOR_SCALE, --color_scale COLOR_SCALE
                        Scale factor for 'color' stream preview window.
                        Default: 1.0
  -monor {400,720,800}, --mono_resolution {400,720,800}
                        Mono cam res height: (1280x)720, (1280x)800 or
                        (640x)400 - binning. Default: 720
  -monof MONO_FPS, --mono_fps MONO_FPS
                        Mono cam fps: max 60.0 for H:720 or H:800, max 120.0
                        for H:400. Default: 30.0

And of course these are configurable via C++ and Python API (see here for Python example).

Thoughts?

Cheers, Brandon

[mationai]

Excellent, thank you!

[eric-schleicher]

bending the topic. I'm curious what other (mipi) global shutter sensor would be considered for faster than 200FPS? I have a very cool application in mind if 300+ is possible at ~VGA. I have ov9282 standalone and it's not quite fast enough.

[Luxonis-Brandon]

So we have in progress OV7750/OV7251 but it's quite lower resolution:

• OV7251 is capable of capturing VGA (640x480) resolution video at 120 frames per second (fps),
• QVGA (320x240) at 180 fps with binning, and
• QQVGA (160x120) at 360 fps with binning and skipping

So the flow of these global-shutter from OmniVision is: OV7750/OV7251 -> OV9282 (and OV9782 for color equivalent) -> ?

So the '?' could already exist... I'm not sure if OmniVision has made the 'next gen' version of the OV9282, but if they have, I would guess that it could do VGA at over 300 FPS.

Since the OV9282 has been out for what seems like forever, I would expect this soon if it's not already out. It may be out and it could be that my Google skills are not sufficiently strong. Anyway, here are the sensors for which we theoretically have driver support (but many are untested):

• IMX214
• IMX283
• IMX334
• IMX378
• IMX477
• OV9282
• OV9281
• OV9782
• OV7750/OV7251
• IMX412
• SC2232H
• OV2735
• OV5645
• IMX390
• SC5335
• IMX363
• SC8238
• OV12895

I think I had reviewed them for global shutter capability, but there's a chance I missed one.

Thoughts?

Thanks, Brandon

[Luxonis-Brandon]

As an update, this looks quite doable from the first look. We just need to crank out some critical items (like depth alignment, IMU support, object tracker in Gen2, etc.) and then will be able to implement this. So I'm thinking likely April-ish for this.

[bgragavendiran]

I have an Android Device I am trying to interface OV12895 camera drivers, how should I proceed where can I find the C/C++ code and the DTSI file to integrate the Sensor to my android device.

I have forked a AOSP version of the OS to start off with, everything is working so far, just need to integrate this camera