Nvidia GPU features not working

[izevar]

Hello Seem that all GPU related features are not working: GPU decoding GPU encoding Object detection

There is any extra step needed to enable it ?

[ispysoftware]

you might need to get an ffmpeg build with GPU enabled or install a video codec SDK. The GPU stuff is only tested on Windows. Something like: https://hub.docker.com/r/willprice/nvidia-ffmpeg/ Note that RAW recording will always be faster than GPU encode if you are using IP cameras as there is basically no encode step.

[doitandbedone]

@izevar did you try the above steps? Did they work? Could you please give us an update? Thank you!

[stripeyhorse]

im not having any performance issues , but is intel quicksync supported under docker.

btw im using raw and having no issues at all. just trying to make sure it uses as little resources as possible.

[ispysoftware]

we're working on quicksync under docker. Nothing will beat raw for encoding though.

On Tue, Nov 10, 2020 at 5:56 PM stripeyhorse [email protected] wrote:

im not having any performance issues , but is intel quicksync supported under docker.

btw im using raw and having no issues at all. just trying to make sure it uses as little resources as possible.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/doitandbedone/ispyagentdvr-docker/issues/79#issuecomment-724594735, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGDKXIH6EB3FVN73CSOKR3SPEE4ZANCNFSM4TF2GKTA .

[ispysoftware]

@doitandbedone test version here for linux h264 support https://ispyrtcdata.blob.core.windows.net/downloads/Agent_Linux64_H264.zip see server settings and you can set the Codec to H264 on the General tab. Then set the encoder on the recording tab on the camera to use quicksync and if the drivers are installed properly it should use them. See the logs for messages. Specifically on Agent startup if quicksync is going to work there should be a message about it under "Hardware Support"

[doitandbedone]

Perfect. I'll give it a try. Thanks!

On Thu, Nov 12, 2020, 11:38 PM sean tearney [email protected] wrote:

@doitandbedone https://github.com/doitandbedone test version here for linux h264 support https://ispyrtcdata.blob.core.windows.net/downloads/Agent_Linux64_H264.zip see server settings and you can set the Codec to H264 on the General tab. Then set the encoder on the recording tab on the camera to use quicksync and if the drivers are installed properly it should use them. See the logs for messages. Specifically on Agent startup if quicksync is going to work there should be a message about it under "Hardware Support"

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/doitandbedone/ispyagentdvr-docker/issues/79#issuecomment-726577116, or unsubscribe https://github.com/notifications/unsubscribe-auth/AORTFNBCCXDGVX7KVUQ5SLTSPTO6LANCNFSM4TF2GKTA .

[doitandbedone]

Sorry for the delay, been having issues lately with the sever, spent the last few days to put it back up and not lose any of my data phew. So back to almost normal but still need to do a few other things. If everything goes well I should be able to run a test tomorrow. I really appreciate all the help and effort you're putting into this!

[oriol73]

Hi there, I'm running a server with Ubuntu 20.04, Docker 19.03.13, with your AgentDVR container. I've been trying to enable GPU decoding/encoding as well to reduce CPU workload. 5x IP cameras (four RSTP streams at 1080p, one at 720p bring my cpu above 60%). I have one Nvidia Geforce GT710 with nvidia-driver-455.38 installed, nvidia-docker2, docker nvidia/cuda 11.1 , nvtop. All up and running.

If I log into the container via shell and retry dotnet /agent/Agent.dll, I get an ffmpeg output like this:

Init FFmpeg ffmpeg path: /usr/lib/x86_64-linux-gnu Error: Unable to load DLL 'avutil.56': The specified module could not be found. Error: at FFmpeg.AutoGen.ffmpeg.LoadLibrary(String libraryName, Boolean throwException) at FFmpeg.AutoGen.ffmpeg.<>c.<.cctor>b__6_0(String libraryName) at FFmpeg.AutoGen.ffmpeg.<>c.<.cctor>b__6_604(Int32 level) at FFmpeg.AutoGen.ffmpeg.av_log_set_level(Int32 level) at CoreLogic.FFmpeg.SetFfmpegLogging(Int32 loglevel) at CoreLogic.FFmpeg.Init() at CoreLogic.Threads.StartUp() Unable to load DLL 'avutil.56': The specified module could not be found. Install FFMPEG (v4) using the terminal. See the download page on ispyconnect.com for instructions. Killed

I can run some tests for you on my server if this helps us speed up the resolution.

Best regards, oriol

[ispysoftware]

you need to use a build of ffmpeg that has the libraries instead of the static binaries - agent uses the libraries like avutil.56 not the ffmpeg binary file.

also to decode using the gpu you need to select GPU on the decoder options under the video source settings.. and you'd need to use the build of agent that i uploaded up there ^^ somewhere

[doitandbedone]

Yes, that's where I left off as well. Seems the recommended ffmpeg repo has other libraries included. I was using the logs to try and install all the ones it complained about and use the compilation flags for it too. Still a work in progress though. The build is posted in an earlier comment, you will have to modify the current Dockerfile to make use of the new build however. Once the version is ready I'll make it available more publicly.

[ispysoftware]

Do you have it working? It works fine on Windows so in theory should be OK on Docker. As long as the ffmpeg parts are there

[doitandbedone]

Not yet, ffmpeg is proving to be the most challenging part so far, but then again, haven't been able to get back to it, there's two branches I created if you want to take a look.

[ispysoftware]

I'm not 100% certain you need to rebuild it. If it can save to mp4 using the h264 codec then it should be able to use the gpu if the codecs are available

[doitandbedone]

So I finally had to time to run it, ran into a few issues (dirvers/cuda version) but all resolved, however, it still prefers software. I do see a warning: [37m[40mWarning: h264 compatible version of ffmpeg must be installed to use H264 encoding Do I need another version, or is this just a friendly warning?

And a few errors/warnings:

[37m[40mParking: Failed to create specified HW device.

[32m[40m[h264 @ 0x14c03403ba40] A hardware frames or device context is required for hardware accelerated decoding.
[32m[40m[h264 @ 0x14c03403ba40] Failed setup for format vaapi_vld: hwaccel initialisation returned error.

[37m[40mParking: Failed to create specified HW device.

[32m[40m[h264 @ 0x14c03403ba40] A hardware frames or device context is required for hardware accelerated decoding.
[32m[40m[h264 @ 0x14c03403ba40] Failed setup for format vaapi_vld: hwaccel initialisation returned error.

I followed setup as instructed:

• Set server settings to Codec H264/MP4/MP3
• Set Recording to quicksync Additionally, I tried going into Video Source (camera settings) and change Decoder to GPU (used FFMPEG as well), but no luck on either.

Full logs: nvidia-docker-ispy-error.zip

[ispysoftware]

if you set LogFFMPEGLevel in the media/xml/config.xml file to 48 ffmpeg should write a whole load of debug to the logs.

On Mon, Nov 23, 2020 at 1:58 PM doitandbedone [email protected] wrote:

So I finally had to time to run it, ran into a few issues (dirvers/cuda version) but all resolved, however, it still prefers software. I do see a warning: �[37m�[40mWarning: h264 compatible version of ffmpeg must be installed to use H264 encoding Do I need another version, or is this just a friendly warning?

And a few errors/warnings:

�[37m�[40mParking: Failed to create specified HW device.

�[32m�[40m[h264 @ 0x14c03403ba40] A hardware frames or device context is required for hardware accelerated decoding.

�[32m�[40m[h264 @ 0x14c03403ba40] Failed setup for format vaapi_vld: hwaccel initialisation returned error.

�[37m�[40mParking: Failed to create specified HW device.

�[32m�[40m[h264 @ 0x14c03403ba40] A hardware frames or device context is required for hardware accelerated decoding.

�[32m�[40m[h264 @ 0x14c03403ba40] Failed setup for format vaapi_vld: hwaccel initialisation returned error.

I followed setup as instructed:

• Set server settings to Codec H264/MP4/MP3 [image: Screen Shot 2020-11-22 at 9 50 03 PM] https://user-images.githubusercontent.com/61026996/99932641-8985bc80-2d0d-11eb-8ade-7ccd5af9adf6.png
• Set Recording to quicksync [image: Screen Shot 2020-11-22 at 9 50 21 PM] https://user-images.githubusercontent.com/61026996/99932658-91ddf780-2d0d-11eb-949e-492d4af91fa4.png Additionally, I tried going into Video Source (camera settings) and change Decoder to GPU (used FFMPEG as well), but no luck on either. [image: Screen Shot 2020-11-22 at 9 51 25 PM] https://user-images.githubusercontent.com/61026996/99932673-9d312300-2d0d-11eb-9ff2-a1dd731fe9b8.png [image: Screen Shot 2020-11-22 at 9 51 32 PM] https://user-images.githubusercontent.com/61026996/99932674-9dc9b980-2d0d-11eb-950c-1c13763fd36c.png

Full logs: nvidia-docker-ispy-error.zip https://github.com/doitandbedone/ispyagentdvr-docker/files/5581268/nvidia-docker-ispy-error.zip

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/doitandbedone/ispyagentdvr-docker/issues/79#issuecomment-731944709, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGDKXNGRRIIZXXSOJZQQGLSRH22FANCNFSM4TF2GKTA .

[doitandbedone]

Woah, after enabling more logs it goes haywire, it gets so overwhelming it starts to freeze the server...

Anyhow, had to kill the container and collect logs afterwards. The container also seems to stay fine, so got the logs from it as well. They seem to be truncated given the huge amount of data. Hope it helps: ffmpeg-loglevel-48.zip

[ispysoftware]

It's missing the connection logs where it complains about creating the hardware context. Stop the camera, make sure its all set up to use the gpu, start it and stop it and then save off the logs.

[ispysoftware]

GPU support isn't available on docker on windows, which is my dev environment :( https://www.edureka.co/community/19378/is-gpu-pass-through-possible-with-docker-for-windows

With docker on linux it seems you have to forward the device to the container for it to be able to access it like: https://github.com/jellyfin/jellyfin/issues/2281#issuecomment-695947737 specifically calling docker with: --device /dev/dri:/dev/dri \

also https://stackoverflow.com/questions/25185405/using-gpu-from-a-docker-container

[doitandbedone]

Yes, I'm using the special nvidia cuda container: https://github.com/NVIDIA/nvidia-docker And I am passing the GPU over, I can actually see it in the container when running nvidia-smi:

+-----------------------------------------------------------------------------+ | NVIDIA-SMI 440.59 Driver Version: 440.59 CUDA Version: 10.2 | 0 GeForce GTX 750 Ti +-------------------------------+----------------------+----------------------+

One thing I noticed however, seems the ffmpeg does not have cuda enabled: Hardware acceleration methods: vdpau vaapi drm opencl

I noticed since I tried using the available libx264: #> ffmpeg -y -vsync 0 -hwaccel cuda -hwaccel_output_format cuda -i 11_TimeLapse_2020-12-03_07-04-43.mp4 -c:a copy -c:v libx264 -b:v 5M output.mp4 Device creation failed: -12. [h264 @ 0x5636e258bd80] No device available for decoder: device type cuda needed for codec h264.

Output: Stream mapping: Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264)) Device setup failed for decoder on input stream #0:0 : Cannot allocate memory

Perhaps ffmpeg needs to be rebuild to enable cuda acceleration?

[ispysoftware]

i've got it working on my linux test box on ubuntu 18, hardware encode h264 and decode using vaapi, needed quite a lot of code changes. Will have an update soon. Testing on OSX/ VP8 etc.

On Thu, Dec 3, 2020 at 11:31 PM doitandbedone [email protected] wrote:

Yes, I'm using the special nvidia cuda container: https://github.com/NVIDIA/nvidia-docker And I am passing the GPU over, I can actually see it in the container when running nvidia-smi. One thing I noticed however, seems the ffmpeg does not have cuda enabled: Hardware acceleration methods: vdpau vaapi drm opencl

I noticed since I tried using the available libx264: #> ffmpeg -y -vsync 0 -hwaccel cuda -hwaccel_output_format cuda -i 11_TimeLapse_2020-12-03_07-04-43.mp4 -c:a copy -c:v libx264 -b:v 5M output.mp4 Device creation failed: -12. [h264 @ 0x5636e258bd80] No device available for decoder: device type cuda needed for codec h264.

Output: Stream mapping: Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264)) Device setup failed for decoder on input stream #0:0 : Cannot allocate memory

Perhaps ffmpeg needs to be rebuild to enable cuda acceleration?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/doitandbedone/ispyagentdvr-docker/issues/79#issuecomment-738080907, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGDKXMOBPI3KPGRP3QOMJTSS6VNRANCNFSM4TF2GKTA .

[ispysoftware]

here's an updated version - use this and set the recorder mode to GPU and the codec in settings to h264 https://ispyrtcdata.blob.core.windows.net/downloads/Agent_Linux64.zip Should have hardware decode and encode using vaapi. Any probs check the logs.

[doitandbedone]

Wow, you beat me to it.. I'll try it, give me a few mins. I did managed to recompile ffmpeg but with minimal codecs. I tried using Jonathon's PPA as a base for all the libraries it installs. I then tried to get ffmpeg 4.3.1 source and recompile with old + new config flags but ffmpeg configure cannot find the reference to the libraries it had installed even though they are there. Anyhow, let me see...

[doitandbedone]

Did you take any extra steps to get it working? Seems it's not using the GPU, however, I turn everything off and rebooted to see starting logs, I see this:

[37m[40mffmpeg path: /usr/lib/x86_64-linux-gnu [37m[40mInitializing devices [37m[40mHardware Decoders: [37m[40mvdpau [37m[40mvaapi [37m[40mdrm [37m[40mopencl [37m[40mCUDA not available [37m[40mcall ffplay -nodisp -f v4l2 -list_formats all /dev/video0 [37m[40m[video4linux2,v4l2 @ 0x14d760000b80] cannot open video device /dev/video0: no such file or directory

I tried a standalone command: ffmpeg -hwaccel vdpau -i /agent/Media/WebServerRoot/Media/video/Mazda\ Parking/13_2020-11-22_21-08-50_542.mp4 /agent/Media/WebServerRoot/Media/output.mp4

Output:

[AVHWDeviceContext @ 0x5594cf789880] Cannot open the X11 display . Device creation failed: -1313558101. [h264 @ 0x5594cf6942c0] No device available for decoder: device type vdpau needed for codec h264. Stream mapping: Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264)) Device setup failed for decoder on input stream #0:0 : Unknown error occurred

I'm thinking compatibility issues with the card? Or somehow missing something that is not letting the card be seen? I do get a good output with nvidia-smi thoug,h as shown before.

[doitandbedone]

So I went back to your comment earlier on passing the devices. I guess internally it's looking for /dev/video0 ... /dev/video<n> So then I found this: https://stackoverflow.com/questions/25185405/using-gpu-from-a-docker-container Recommends to use the following to get your device list: ls -la /dev | grep nvidia Then forward it: --device /dev/<host-device>:/dev/<container-device> Mine spits out:

crw-rw-rw- 1 root root 195, 254 Dec 2 12:05 nvidia-modeset crw-rw-rw- 1 root root 244, 0 Dec 2 12:05 nvidia-uvm crw-rw-rw- 1 root root 244, 1 Dec 2 12:05 nvidia-uvm-tools crw-rw-rw- 1 root root 195, 0 Dec 2 11:37 nvidia0 crw-rw-rw- 1 root root 195, 255 Dec 2 11:37 nvidiactl

I then tried --device /dev/nvidia0:/dev/video0 And the logs now don't spit out the error:

[37m[40mcall ffplay -nodisp -f v4l2 -list_formats all /dev/video0 [37m[40m[video4linux2,v4l2 @ 0x14fb88000b80] ioctl(vidioc_querycap): invalid argument [37m[40mcall ffplay -nodisp -f v4l2 -list_formats all /dev/video1 [37m[40m[video4linux2,v4l2 @ 0x14ce40000b80] cannot open video device /dev/video1: no such file or directory

But as you may have noticed, it then goes onto /dev/video1. Not too sure about that "invalid argument" though. How should devices be forwarded? Not too sure how to correlate them, what do you recommend?

P.S.: Do note that the devices are being forwarded by nvidia onto the container without the need of --device, the print was in fact from inside the container (and it prints the same on the host machine), they are just referenced differently within agent's commands/libs.

[ispysoftware]

i've been concentrating on getting it working on linux, should work on docker as long as the devices are forwarded into the container.

maybe this will help: https://github.com/doitandbedone/ispyagentdvr-docker/issues/100#issuecomment-733405196

[doitandbedone]

Right, I mean at the end of the day docker container is running ubuntu 18.04. The question again is what the devices ffplay is expecting in what order. The nvidia runtime forwards with the naming printed above. But ffplay seems to be using /dev0, /dev1 not too sure what those are supposed to be mapped to. nvidia-uvm, nvidia-actl, nvidia-0, etc

[ispysoftware]

Did you try "I just launched the container with --device /dev/dri:/dev/dri "?

[doitandbedone]

Scratch what I said earlier, that's probably some other error, I found that dev/device-N it's just for any connected device. As for the forwarding, yes, I also tried the manual command: ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -i /agent/Media/WebServerRoot/Media/video/Mazda\ Parking/13_2020-11-22_21-08-50_542.mp4 /out.mp4

Output:

[AVHWDeviceContext @ 0x558fe3c59ac0] libva: va_getDriverName() failed with unknown libva error,driver_name=(null) [AVHWDeviceContext @ 0x558fe3c59ac0] Failed to initialise VAAPI connection: -1 (unknown libva error). Device creation failed: -5. [h264 @ 0x558fe3afc300] No device available for decoder: device type vaapi needed for codec h264. Stream mapping: Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264)) Device setup failed for decoder on input stream #0:0 : Input/output error

Seems there's some driver issues? Still looking around...

Edit: Just wanted to add what gets forwarded when adding the /dev/dri/:

drwxr-xr-x 2 root root 80 Dec 4 14:20 . drwxr-xr-x 6 root root 480 Dec 4 14:20 .. crwxrwxrwx 1 99 users 226, 0 Dec 4 14:20 card0 crwxrwxrwx 1 99 users 226, 128 Dec 4 14:20 renderD128

[ispysoftware]

The other post mentions:

apt update
apt install mesa-va-drivers

[ispysoftware]

did that resolve it?