Gluetun process CPU usage

[giovannipollo]

Is this urgent?

No

Host OS

Ubuntu Server 20.04 LTS

CPU arch

x86_64

VPN service provider

Mullvad

What are you using to run the container

Portainer

What is the version of Gluetun

The problem happens in all versions

What's the problem 🤔

The problems happens in all releases, it is not a problem of the latest release. There are a lot of processes that I can see with htop that consumes a lot of CPU. I wanted to know if it normale or not.

Here a picture:

The column on the left is the CPU usage. As you can see there are some process that consumes up to 130% of CPU

Share your configuration

---
version: "2.1"
services:
  qbittorrent:
    image: lscr.io/linuxserver/qbittorrent
    container_name: qbittorrent
    network_mode: service:gluetun
    depends_on:
      gluetun:
        condition: service_healthy
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/Rome
      - WEBUI_PORT=8080
    volumes:
      - /xxxx:/xxxx
      - /xxx:/xxxx
    restart: unless-stopped
  gluetun:
    image: qmcgaw/gluetun
    container_name: gluetun_torrent
    ports:
      - 8080:8080
      - 6881:6881
      - 9696:9696
      - 6881:6881/udp
      - 8096:8096
      - 8920:8920
    cap_add:
      - NET_ADMIN
    environment:
      - VPN_SERVICE_PROVIDER=mullvad
      - VPN_TYPE=wireguard
      - WIREGUARD_PRIVATE_KEY=xxxxx
      - WIREGUARD_ADDRESSES=xxxx
      - SERVER_CITIES=xxxxx
      - FIREWALL_OUTBOUND_SUBNETS=xxxx
    restart: unless-stopped
  prowlarr:
    image: lscr.io/linuxserver/prowlarr:develop
    container_name: prowlarr
    network_mode: service:gluetun
    depends_on:
      gluetun:
        condition: service_healthy
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/Rome
    volumes:
      - /xxx:/xxxx
    restart: unless-stopped
  emby:
    image: lscr.io/linuxserver/emby:latest
    container_name: emby
    network_mode: service:gluetun
    depends_on:
      gluetun:
        condition: service_healthy
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/Rome
    volumes:
      - /xxx:/xxx
      - xxxx:/xxxx
    devices:
      - /dev/dri:/dev/dri #optional
    restart: unless-stopped

[qdm12]

Does it use the kernelspace or userspace wireguard (check your logs)? I'd guess it uses the userspace one hence the usage (especially if you download stuff actively). There are multiple processes since it's multi-threaded.

On the other hand, the kernelspace may not show as cpu usage from /gluetun-entrypoint, although it would likely use the same amount of cpu (a bit less) but in the kernel.

[qdm12]

If you want to dig into it, you can check out this new wiki page: https://github.com/qdm12/gluetun/wiki/Profiling it's relatively straight forward to setup and quite fun too 😉 Mine is running on kernelspace so I don't think I can spot the performance impact using this.

[giovannipollo]

Does it use the kernelspace or userspace wireguard (check your logs)? I'd guess it uses the userspace one hence the usage (especially if you download stuff actively). There are multiple processes since it's multi-threaded.

On the other hand, the kernelspace may not show as cpu usage from /gluetun-entrypoint, although it would likely use the same amount of cpu (a bit less) but in the kernel.

As you suggsted I think I am running the userspace. In fact, from the portainer log, I can see this line:

INFO [wireguard] Using userspace implementation since Kernel support does not exist

In order to try the kernelspace and see if performance improves, how can I enable it? Thanks

[qdm12]

It's picked up automatically if it's available in your kernel. On Ubuntu sudo apt install wireguard should do I think. I would be curious if you find any improvement so feel free to comment back with your findings.

[giovannipollo]

Ok I checked and it is installed. However, as said here (I think you can merge the discussion into a single one), when trying to use the kernelspace it does not work properly

[qdm12]

You can try https://github.com/qdm12/gluetun/pull/1268#issue-1472719470 and report with your logs, if Wireguard is not detected in the kernel.

Side note, you can now set WIREGUARD_IMPLEMENTATION to auto | userspace | kernelspace, although it will likely error out with kernelspace if it's not found to be supported.

[ioqy]

I have the same problem and after a restart of the machine I only needed to restart the docker container once after boot for gluetun to be able to use the kernel module and then everything was fine. But since today the kernel implementation inside the container couldn't be used no matter what.

After a bit of search I found that linuxserver/docker-wireguard uses modprobe to load the wireguard kernel module. So I added the SYS_MODULE capability, /lib/modules as a read-only volume mount and the following shell script as the entrypoint to my compose file:

#!/bin/sh

modprobe wireguard

/gluetun-entrypoint

And since then I did multiple reboots of the machine and didn't have a single time that the wireguard kernel module could not be loaded by gluetun.

I also tried it without modprobe and only the SYS_MODULE capability and the /lib/modules volume mount, but gluetun didn't load the kernel module on it's own.

Maybe the loading of the kernel module can be added to gluetun?

[qdm12]

But since today the kernel implementation inside the container couldn't be used no matter what.

What do you mean? With the latest image? Does it work with v3.35.0?

Maybe the loading of the kernel module can be added to gluetun?

Yes totally, working on it

[ioqy]

I tried it with both v3.35.0 and v3.34.3. Up until yesterday gluetun couldn't use the kernel module on the first start of the container after a reboot of the host machine. I use gluetun for maybe close to a year now and have had this problem from the start. So it's not a new thing.

I had a job that would restart the gluetun container after boot of the host machine. And up until yesterday that worked fine. But yesterday gluetun couldn't use the kernel module no matter how many times I restartet the container.

Did a little search and found that linuxserver/docker-wireguard must have had the same problems and uses modprobe in shell script to load the kernel module before starting wireguard. So I then added the above script with modprobe as the entrypoint for my gluetun container and did a couple of reboots of the host machine and gluetun could use the kernel module after every single reboot at the first startup of the container.

[qdm12]

Please try image qmcgaw/gluetun:pr-1741 after a reboot to see if it loads the kernel module properly. You can use LOG_LEVEL=debug which will debug log if there is an error loading the Kernel module.

I went the hard way (for whatever fun and obscure reason) to have it coded in Go, so it may or may not work the first try!

The mechanism is now as follows:

1. Check for the Wireguard family. If it exists, Wireguard is supported.
2. If not, trying loading the wireguard kernel module. Note some systems (like WSL) have the Wireguard family but no wireguard kernel module available to load, that is why this is step 2 and not step 1. If this fails, Wireguard is assumed to be not supported.
3. Check again (really a double check) the wireguard family is now present. If it exists, Wireguard is supported. This should always be true really unless the wireguard kernel module is messed up somehow.

Feel also free to comment on #1741 directly.

[qdm12]

Done in #1741