jetson-utils icon indicating copy to clipboard operation
jetson-utils copied to clipboard

python: add initial cudaFromGstSample support

Open aconchillo opened this issue 5 years ago • 7 comments

This patch adds initial bindings support for GStreamer. It currently only provides a function cudaFromGstSample.

cudaFromGstSample expects a GstSample as a single argument. The GstSample needs to be in NV12 format and stored in host memory (this means there's no zero-copy). The function will then copy the frame to CUDA memory and convert it to RGBA (float4) directly in the GPU.

A usage example could be:

def on_new_sample(appsink):
    sample = appsink.emit('pull-sample')

    img, width, height = jetson.utils.cudaFromGstSample(sample)

    # classify the image
    class_idx, confidence = net.Classify(img, width, height)

When using GStreamer for, let's say, decoding a video encoded in H.264, it is very likely that OpenMAX GStreamer plugins are used (i.e. omxh264dec). This will create a video/x-raw(memory:NVMM) frame that is mapped into NVMM memory (i.e. DMA buffer). Unfortunately, it's not straight forward to convert NVMM memory to CUDA, that's why cudaFromGstSample expects the frame to be stored in the host memory. To copy an NVMM mapped frame to host memory, your pipeline needs to include the nvvidconv plugin which will convert an NVMM mapped frame to a regular video/x-raw frame.

aconchillo avatar Aug 08 '19 07:08 aconchillo

nice!

do you have an example including the GstSample ? which package do you use for that on a nano?

thanks.

blitzvb avatar Aug 13 '19 13:08 blitzvb

Thanks for the contribution @aconchillo!

If you have the code of reading video using this (presumably using appsink), would like to merge it all.

dusty-nv avatar Aug 16 '19 18:08 dusty-nv

New examples PR https://github.com/dusty-nv/jetson-inference/pull/389

aconchillo avatar Aug 17 '19 05:08 aconchillo

Added a new VideoLoader class. Not sure if I also need to added to the Jetson.Utils package, I don't know the difference.

aconchillo avatar Aug 17 '19 05:08 aconchillo

There was an issue with nvv4l2decoder negotiating caps with videorate. This is now fixed and works with JetPack 4.2.1. nvv4l2decoder now has a higher ranking (therefore chosen) than omxh264dec.

aconchillo avatar Aug 27 '19 05:08 aconchillo

Renamed VideoLoader to VideoSource since I'm planning to add a VideoSink at some point.

aconchillo avatar Sep 03 '19 03:09 aconchillo

Added a VideoSink class useful to create video files. Also added GstBuffer to numpy array conversions.

aconchillo avatar Nov 05 '19 07:11 aconchillo