moby
vpnkit.exe eating my RAM
This happened after updating to 18.* branch. At this moment I have 18.03.0-ce-rc4-win57 (16511) build and my vpnkit.exe steals gigabytes of RAM in 2-3 hours with 2 containers running.
Like this:
Or even like this:
I think it should never consume about 8 GBs of my RAM.
I have this issue as well. vpnkit.exe claims memory in proportion to the amount of network activity into and out of the docker containers, and never releases it.
Edit: the latest version 18.03.0-ce-win58 (16761) may have resolved the issue. Fingers crossed.
I've updated my Docker to 18.03.0-ce-win58 (16761) two hours ago - and problem is still here:
Thanks for your reports.
In order to make progress with this issue I need some reproduction steps. Could you provide a docker-compose.yml (or similar) and instructions to reproduce the problem?
After trying a bit, it seems opening an HTTPS connection to a server on our internal network triggers the bug. The same doesn't apply to external, public servers (i.e. docker.com) nor other Docker instances.
@laarmen thanks for the update. Could you trigger the bug and then upload a diagnostic report? I'd like to take a look at the logs.
See https://github.com/laarmen/VpnKitPoC for the code. How can I do the diagnostic report thing?
On Windows there should be a whale-shaped icon in the system tray. After right clicking on it there should be a menu item called something like "Diagnose and Feedback". Clicking on this should take you to a dialog where diagnostics are uploaded and assigned a unique id. If you quote the id in the ticket then I can download the logs and take a look.
(Sorry I couldn't give more precise instructions but I don't have a Windows machine to hand)
On Fri, Mar 30, 2018 at 2:19 PM, Simon Chopin [email protected] wrote:
See https://github.com/laarmen/VpnKitPoC for the code. How can I do the diagnostic report thing?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/moby/vpnkit/issues/371#issuecomment-377522486, or mute the thread https://github.com/notifications/unsubscribe-auth/AAMHul2HIQrFUI_nFNH_aLxIOT-jhgCTks5tjjDlgaJpZM4S6LDK .
I'm not entirely sure this was an instance of the bug as I was still under the 300MB bar of RAM used by vpnkit, but it was consistently climbing. I'll upload another report if I get to the "eat-my-RAM" levels later.
ID: 5E3BFA7A-FF8F-4077-8583-773FF79518CC/2018-03-30_18-33-45
In case that's useful, I just stopped all the docker containers on my workstation, waited a few minutes, and the vpnkit process sits at 700MB. I uploaded a second diagnosis, see 5E3BFA7A-FF8F-4077-8583-773FF79518CC/2018-03-30_19-09-21
This time on my home computer and network, same code except that the target (on local network) is using plain http (no SSL), the memory grew to 1.5GB.
ID: D25DA2F3-2F67-42BA-A292-78A39BCBAEC4/2018-03-30_20-34-58
We are having same issue. This is happening to us in under one day (although we are using an app that generates a lot of network traffic). So we currently have to bounce docker once per day.
Same thing for me - I'm running a single Node.js process that downloads files from the web over HTTP (text and binary) - some 25-30K files, ~1GB in volume, about 100KB/s. VPNKit process consumes all available RAM within hours (I've had it consume up to 9GB of RAM, even though the overall limit for Docker itself is 2GB).
Same here. Win 10 x64, docker version:
Client:
Version: 18.03.0-ce
API version: 1.37
Go version: go1.9.4
Git commit: 0520e24
Built: Wed Mar 21 23:06:28 2018
OS/Arch: windows/amd64
Experimental: false
Orchestrator: swarm
Server:
Engine:
Version: 18.03.0-ce
API version: 1.37 (minimum version 1.24)
Go version: go1.9.4
Git commit: 0520e24
Built: Wed Mar 21 23:21:06 2018
OS/Arch: windows/amd64
Experimental: true
vpnkit.exe currently at 7GB, constantly climbing:
0 containers running. Uploaded a diagnostic, id: 29542F91-6441-4210-934F-DB948F4EF0EF/2018-04-04_16-34-22
UPDATE 24/04/2018: As suggested below, adding vpnkit commit sha:
bazzilic@CSLRF21 $ & "C:\Program Files\Docker\Docker\resources\vpnkit.exe" --debug --ethernet foo
vpnkit.exe: [INFO] Setting handler to ignore all SIGPIPE signals
vpnkit.exe: [INFO] Version is 7c425f691978cb4a708ccc295dd331eae5cebc85
Here's a simple docker-compose.yaml that can reproduce the issue. If you watch memory usage on vpnkit.exe when this is running, it climbs by almost 1M every time the wget runs.
version: '3'
services:
eat-memory:
image: busybox
entrypoint: sh
command:
- -c
- |
while true; do
echo Getting docker.com...
wget -qO/dev/null https://www.docker.com
sleep 5
done
In my case it climbs even if there’s no activity related to docker at all. At least, nothing explicit.
Even with all windows and linux containers stopped, the memory usage is constantly around 1.5 GB on my machine...
I have this same issue occurring with 3 containers that are doing a large amount of WAN activity. If left to run over a week this will consume all the available RAM and leave the system in a unstable state. My only work around is restarting docker regularly.
Cross link forum entry of many folks with the same vpnkit memory issue: https://forums.docker.com/t/vpnkit-uses-all-free-memory/48558/12 For me I suspect the behavior appeared with the 16762 build - never noticed this before but wasn't looking until it exhausted my memory for one simple nginx container.
I am having the same issue.
Docker Version 18.03.0-ce-win59 (16762)
Windows Server 2016 with 32GB of Memory
Limit docker to 10gb of memory and vpnkit.exe uses up to 16GB in 24 hours.
It either crashes or I have to restart Docker
I hate to post a me too but, me too: Docker Version 18.03.0-ce-mac60 (23751) Channel: stable 6ddfc0f1d3 OSX 10.11.6 (16GB Ram) Running one talkative (http outgoing requests only) app via docker compose
"me too" Left a couple (mostly idle) containers running over the weekend came back to 4GB used by vpnkit and a cranky system as that's what I had left..
Client: Version: 18.03.0-ce API version: 1.37 Go version: go1.9.4 Git commit: 0520e24 Built: Wed Mar 21 23:06:28 2018 OS/Arch: windows/amd64 Experimental: false Orchestrator: swarm
Server: Engine: Version: 18.03.0-ce API version: 1.37 (minimum version 1.12) Go version: go1.9.4 Git commit: 0520e24 Built: Wed Mar 21 23:14:32 2018 OS/Arch: linux/amd64 Experimental: false
We've rolled back to 17.12.0-ce-win47 2018-01-12 and are no longer seeing this issue.
Had an idea to compare versions so that we can isolate the vpnkit version that started failing.
PS C:\Program Files\Docker\Docker\Resources> .\vpnkit.exe --version
%%VERSION%%
PS C:\Program Files\Docker\Docker\Resources>
Sigh.
.\vpnkit.exe --debug --ethernet foo will spit out a git sha:
PS C:\Program Files\Docker\Docker\Resources> .\vpnkit.exe --debug --ethernet xxx
vpnkit.exe: [INFO] Setting handler to ignore all SIGPIPE signals
vpnkit.exe: [INFO] Version is eb91fd8319abdfcaf87a1839e46b7ce0577b68fc
...
That's for the current Version 18.04.0-ce-rc2-win61 (17070). It corresponds to the most recent commit here.
@tsasioglu What does your 17.12.0-ce-win47 report?
Is there an easy way to roll back to 17.12.0-ce-win? As it is, 18 is completely unusable for me. I have to restart every 90 min because vpnkit uses +90% of my memory
@imarotte Yes, you can download 17.12.0-ce-win47 here: https://docs.docker.com/docker-for-windows/release-notes/#docker-community-edition-17120-ce-win47-2018-01-12. I rolled back two of our staff to that version yesterday. The steps I tool were:
- Uninstall the existing version of Docker for Windows (this removes the VM image - you might want to back it up from the default location of
C:\Users\Public\Documents\Hyper-V\Virtual hard disks\MobyLinuxVM.vhdxand restore after the reinstall) - Restart Windows (might be optional)
- Run the installer from the link provided
- I had to restart Windows again because the
dockercmdlet wasn't working in Powershell
- I had to restart Windows again because the
- Re-add to swarm if appl.
That is the simplest way, although you may end up losing your containers if the VM image is incompatible.
Thanks @imarotte for linking the issue from the docker tracker. When I've been watching mine, it's been rapidly climbing to around 1GB, on a 4GB machine (lol) then backing off, then reclimbing. There's a GIF watching memory usage in Task Man attached to the link in docker/for-win 1932. I have had the machine go completely unresponsive twice in the last week, though, which is highly abnormal, and could be due to disk thrashing caused by memory exhaustion. I haven't yet bumped my Docker version back, because I was hoping that this would get fixed rapidly, and I have enough problems with going forward through Docker versions, that I don't want to see what kind of Hellgate I can open by trying to go backwards. It definitely occurred when upgrading to 18.03, though. I don't know what version I had before, specifically, as I hadn't been paying attention, because I didn't have problems :-)
I do run a nginx in docker that redirects traffic to a few other containers as well as services that run natively on the bare metal. As an aside, I do intend to put more RAM in the box, but so far, it hasn't really presented a problem to me, except that I can't force a docker restart without logging out or rebooting first.
Just picked up the update to 18.03.1, and after about 15 minutes of runtime, vpnkit is hanging out at 13.7 to 14.0MB . . . so.. i'll keep an eye on it, but it seems to be fixed?