vk_video_samples icon indicating copy to clipboard operation
vk_video_samples copied to clipboard

Published 20 hours ago • verified publisher icon nvpro-samples

Remove FFMPEG dependency and use GStreamer instead

Open dabrain34 opened this issue 2 years ago • 8 comments

I'm planning to add GStreamer capabilities to replace the FFMpeg parser in https://github.com/nvpro-samples/vk_video_samples/blob/de54a1df626828c47e265c65d6676a3f083809a1/vk_video_decoder/demos/vk-video-dec/AppDecVulkanFrame/VulkanVideoProcessor.cpp#L49

Do you prefer an option and keep FFMpeg or just like drop FFMpeg for real ?

dabrain34 avatar Oct 04 '22 13:10 dabrain34

See https://github.com/nvpro-samples/vk_video_samples/pull/21

dabrain34 avatar Oct 04 '22 13:10 dabrain34

Do you prefer an option and keep FFMpeg or just like drop FFMpeg for real ?

I would still like to keep the FFMPEG as an option, please, since we still use it as video demuxer with the NV parser.

zlatinski avatar Oct 17 '22 17:10 zlatinski

no problem at all to keep the 2. Its just it would be easier to have one library for all.

dabrain34 avatar Oct 17 '22 17:10 dabrain34

I was looking into GStreamer and it seems like a good alternative to FFMPEG, so I would like to see this PR merged to the main branch at some point too. Anyway the main branch doesn't even work at all right now

BattleAxeVR avatar Feb 01 '23 05:02 BattleAxeVR

We could also use https://github.com/Igalia/ESExtractor

dabrain34 avatar Apr 28 '23 18:04 dabrain34

I would kill to find a video decoder that can also output the motion vectors. That's super useful for using video streams in videogames, if you're using things like DLSS. FFMPEG CPU decoder is supposed to have such a feature but that was ages ago I don't think was ever merged.

BattleAxeVR avatar Apr 28 '23 22:04 BattleAxeVR

Hi @BattleAxeVR, Can you please try https://developer.nvidia.com/optical-flow-sdk? We are still considering a Vulkan video extension that will output the motion vectors, but I'm not sure if this extension will be done this year.

zlatinski avatar Mar 28 '24 22:03 zlatinski

Thanks for the tip, I noticed that project a while back and how it would be suitable, however I've already implemented a working pipeline for vulkan video with explicit MVs stored in a separate video file.

I mostly needed the MVs as inputs to DLSS, which I've discovered can be exploited to produce some rather neat special FX like mirages or optical lensing / hot air shimmering the scene behind it:

image

image

BattleAxeVR avatar Mar 29 '24 05:03 BattleAxeVR