[SDK Feature-Request] rerun as (default) visualizer for oak.visualize()

[Erol444]

Start with the why:

OpenCV approach for visualization is quite limited - it doesn't easily support graphs (eg. IMU logging) or pointclouds (would need open3d) and has some strange quirks (example: cv2.waitKey(1) takes 15ms on Windows, cv2.imshow() has to be run in main thread...).

Rerun is a new visualization SDK/program that is fast and offer many more options.

Move to the what:

At oak.visualize(), have another parameter specifying visualizer. Add rerun as an option (along cv2). If rerun, create viewer and instead of cv2.imshow, log frames/imus/pointclouds (when supported) with rerun python SDK.

Here's initial demo for rerun + depthai.

Move to the how:

To support multiple visualizers, we would likely need to change the SDK architecture of the codebase a bit, eg. abstract visualizer which cv2/rerun visualizers would build upon, and that would have function to visualize frame / imu msgs / pointclouds / etc.

We should move queue logic from XoutBase classes (eg. XoutFrames) to visualizers - as we added the queues so the main thread could act upon new packets; either visualize them via imshow(), or call a callback where the user could use imshow().

So after creating a packet, instead of adding them to queue, SDK (XoutBase) should either pass the packet to the visualizer or call the callback (specified with oak.callback()) with it.

Also, if we don't use cv2 for visualization, we can skip cv2.waitKey(1) in the oak.poll() function.

[brmarkus]

@Erol444 do you already know and have decided (and checked with the authors of "Rerun") under which license you will make this new dependency available to the users? (They seem to still work on their commercial license, according to "https://www.rerun.io/pricing")

[Erol444]

Hi @brmarkus , We are targeting the OSS project (see here) via either MIT/Apache, and will (potentially) share any changes back to their main repo. From what I understand, the commercial license will be for cross-team collaboration, which we will not target, but our users might still be able to use those. Thoughts? Thanks, Erik

[brmarkus]

Sounds good, thanks for the feedback.

[HopeCollector]

I found that when using both oak.record and oak.visualize, the main thread gets stuck in cv2.imshow