Please add support for the HAP codec

[mabrodeur]

Taken form the HAP github page : "[The new HAP video] codec is designed with the goal of playing back as many movies as possible on hardware with fast hard drives, particularly in situations where existing codecs reach the limits of the CPU to decode frames."

"The simplest way to add Hap support to your application is to use QuickTime with the Hap QuickTime codec installed, making a custom request to receive S3TC frames which you then process with OpenGL or DirectX. Discussion and sample code is available in the Hap QuickTime Playback Demo.

If you need to parse raw Hap frames yourself, source code and a specification document are available as part of this project."

It would be really awesome to be able to play this codec in processing as it would greatly improve the performances...

[CaseyJNova]

I would second this!!

[codeanticode]

Need to test the latest beta with a sample HAP video:

https://vdmx.vidvox.net/blog/more-about-the-hap-video-codecs-free-samples-momo-middlman

Hopefully GStreamer 1.16 supports it :-)

[cacheflowe]

I'll third this feature request 🙋‍♂

@codeanticode I just tested several of those HAP example videos with 2.0 beta4 on Windows 10. No errors, but I only got audio - the Movie has a size of 0x0. isLoaded() and hasBufferSink() are false, but isPlaying() is true, and time() works as expected.

My Non-HAP videos play properly and smoothly with the latest library. I was running (2) 4k videos at 50fps. This feels like a 10x performance improvement over the 2.0 library. Very nice work on all of this!

If there's any suggestions on how to help add HAP support, I'd like to contribute.

It looks like gstreamer's HAP support is inside the FFMPEG plugin. Is this possible to add to the updated Video system?

[codeanticode]

@cacheflowe Great to hear that performance is much better!

Regarding HAP support, the beta bundles a fairly complete version of the latest GStreamer (1.16.0) but I removed restricted codecs and a few other packages to reduce size. Not sure if HAP is part of those. You could do a full system install of GStreamer (installers for windows and mac are available here), and the library will override the bundled GStreamer by the system one. Let me know if this enables HAP playback.

[cacheflowe]

@codeanticode I installed both the gstreamer runtime & development versions with all options enabled. It looks like the new video library successfully switched to my local installation 👍

Non-HAP video still plays properly, but now the HAP video throws the error below. Gstreamer can't load a bunch of plugins, but I can confirm they exist in my local installation.

(javaw.exe:26576): GStreamer-WARNING **: 11:11:29.627: Failed to load plugin 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgsta52dec.dll': 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgsta52dec.dll': The specified module could not be found.

(javaw.exe:26576): GStreamer-WARNING **: 11:11:29.630: Failed to load plugin 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstamrnb.dll': 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstamrnb.dll': The specified module could not be found.

(javaw.exe:26576): GStreamer-WARNING **: 11:11:29.633: Failed to load plugin 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstamrwbdec.dll': 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstamrwbdec.dll': The specified module could not be found.

(javaw.exe:26576): GStreamer-WARNING **: 11:11:29.636: Failed to load plugin 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstassrender.dll': 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstassrender.dll': The specified module could not be found.

(javaw.exe:26576): GStreamer-WARNING **: 11:11:29.646: Failed to load plugin 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstdtsdec.dll': 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstdtsdec.dll': The specified module could not be found.

(javaw.exe:26576): GStreamer-WARNING **: 11:11:29.649: Failed to load plugin 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstdvdread.dll': 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstdvdread.dll': The specified module could not be found.

(javaw.exe:26576): GStreamer-WARNING **: 11:11:29.654: Failed to load plugin 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstges.dll': 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstges.dll': The specified module could not be found.

(javaw.exe:26576): GStreamer-WARNING **: 11:11:29.662: Failed to load plugin 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstlibvisual.dll': 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstlibvisual.dll': The specified module could not be found.

(javaw.exe:26576): GStreamer-WARNING **: 11:11:29.664: Failed to load plugin 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstmms.dll': 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstmms.dll': The specified module could not be found.

(javaw.exe:26576): GStreamer-WARNING **: 11:11:29.672: Failed to load plugin 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstresindvd.dll': 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstresindvd.dll': The specified module could not be found.

(javaw.exe:26576): GStreamer-WARNING **: 11:11:29.675: Failed to load plugin 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstrtmp.dll': 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstrtmp.dll': The specified module could not be found.

(javaw.exe:26576): GStreamer-WARNING **: 11:11:29.680: Failed to load plugin 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstsoundtouch.dll': 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstsoundtouch.dll': The specified module could not be found.

(javaw.exe:26576): GStreamer-WARNING **: 11:11:29.689: Failed to load plugin 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstvoaacenc.dll': 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstvoaacenc.dll': The specified module could not be found.

(javaw.exe:26576): GStreamer-WARNING **: 11:11:29.693: Failed to load plugin 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstwebrtcdsp.dll': 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstwebrtcdsp.dll': The specified module could not be found.

(javaw.exe:26576): GStreamer-WARNING **: 11:11:29.695: Failed to load plugin 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstx264.dll': 'D:\gstreamer\1.0\x86_64\lib\gstreamer-1.0\libgstx264.dll': The specified module could not be found.

Processing video library using GStreamer 1.16.0

[codeanticode]

Do you get those errors only when trying to play the HAP video?

[cacheflowe]

I thought I was, but no - These errors were happening for all videos.

I uninstalled and reinstalled the local gstreamer installation, and now the warnings are gone 🤷‍♂ The behavior seems to match the bundled version in 2.0 beta4: no video, but will it play the audio track in a HAP file.

[codeanticode]

@cacheflowe could you share some sample HAP clips for testing? Thank you!

[cacheflowe]

@codeanticode it looks like Vidvox moved the example pack to here:

https://vidvox.net/download/SamplePackOneHap1080p.zip