HWACCEL + tensorflow without leaving GpuRam

[BlueMagma2]

I would like to decode a video using complex_filter and nvdec hardware acceleration. To have it be as fast as possible I would like the decoded frame to be passed to tensorflow directly without leaving the gpu ram so that we don't spend time memcpy-ing the frame from gpu memory to cpu memory and back.

Is it possible using ffmpeg-python ?

[page200]

In the "Experimental" section of TensorFlow, there's the barely documented/commented tfio.experimental.ffmpeg.decode_video() -> ffmpeg_ops.io_ffmpeg_decode_video() -> libtensorflow_io_ffmpeg_4.2.so -> ...

PyTorch seems more promising. TorchAudio (a library for audio and signal processing with PyTorch) talks here about video:

Using NVIDIA’s GPU decoder and encoder, it is also possible to pass around CUDA Tensor directly, that is decode video into CUDA tensor or encode video from CUDA tensor, without moving data from/to CPU.

The bottom of that page leads to tutorials on video decoding and encoding.

Keep us posted how that goes.