Question about jupyterlab / jetcam

[GeneralFox]

Hi, first of all thanks a lot for this repo. I'm a newbie in this field and it's good to understand.

I'm trying to do some experiment with jupyterlab notebook but I haven't found a way to "convert" the my-detection sample in a good way. For what I have understood the display.IsOpen() use the OpenGL / CV library and need a real monitor, so I'm trying to use the jetcam module and ipywidgets/IPython but I haven't figure out yet how to render the detection overlay.

I'm doing that on the jupyterlab notebook:

1. Import and configure the camera:

from jetcam.csi_camera import CSICamera
camera = CSICamera(width=320, height=240)

2. Display the first image:

image = camera.read()
import ipywidgets
from IPython.display import display
from jetcam.utils import bgr8_to_jpeg
image_widget = ipywidgets.Image(format='jpeg')
image_widget.value = bgr8_to_jpeg(image)
display(image_widget)

3. Load the detect net:

import jetson.inference
import jetson.utils
net = jetson.inference.detectNet("ssd-mobilenet-v2", threshold=0.5)

4. Loop:

camera.running = True
def update_image(change):
    image = change['new']
    detections = net.Detect(jetson.utils.cudaFromNumpy(image), 320, 240)
    image_widget.value = bgr8_to_jpeg(image)
camera.observe(update_image, names='value')

The image is correctly refreshed but I was not able to place the overlay with detection on the image. For what I have understood the net.Detect take as input a PyCapsule objet and cudaFromNumpy() should do the conversion but maybe I'm wrong.

Any ideas ?

[jiefengsun]

@GeneralFox Hello, do you have any update for this? Thank you!

[dusty-nv]

@jiefengsun I think it needs to be like this instead:

camera.running = True
def update_image(change):
    image = change['new']
    cuda_img = jetson.utils.cudaFromNumpy(image)
    detections = net.Detect(cuda_img, 320, 240)
    jetson.utils.cudaDeviceSynchronize()
    image = jetson.utils.cudaToNumpy(cuda_img)
    image_widget.value = bgr8_to_jpeg(image)
camera.observe(update_image, names='value')

I've also added more interoperability interfaces for jetson_utils.cudaImage that may make this easier/faster: https://github.com/dusty-nv/jetson-inference/blob/master/docs/aux-image.md#array-interfaces

I've also added WebRTC support for streaming video headlessly to browser, and it will give you higher resolution/framerate and lower latency than Jupyter would: https://github.com/dusty-nv/jetson-inference#webapp-frameworks