Cannot access the VPN network inside a container

[devantoine]

trafficstars

Expected behavior

When I'm connected through a VPN I should be able to access the VPN network inside a container.

Actual behavior

When I'm connected through a VPN I cannot access the VPN network, either by hostname or by IP.

Information

Diagnostic ID: FD552713-528B-478F-9BA6-8A6528AA18A4

Docker for Mac: version: 18.03.0-ce-mac60 (dd2831d4b7421cf559a0881cc7a5fdebeb8c2b98)
macOS: version 10.13.3 (build: 17D102)
logs: /tmp/FD552713-528B-478F-9BA6-8A6528AA18A4/20180418-152905.tar.gz
[OK]     vpnkit
[OK]     vmnetd
[OK]     dns
[OK]     driver.amd64-linux
[OK]     virtualization VT-X
[OK]     app
[OK]     moby
[OK]     system
[OK]     moby-syslog
[OK]     kubernetes
[OK]     files
[OK]     env
[OK]     virtualization kern.hv_support
[OK]     osxfs
[OK]     moby-console
[OK]     logs
[OK]     docker-cli
[OK]     disk

Steps to reproduce the behavior

1. Connect to the VPN (I'm not sure which one it is, it's the client's VPN and I'm using the Pulse Secure client)
2. Start docker-compose
3. Log into a container
4. Try to connect to one of the VPN network machine, using SSH, either by hostname or by IP
5. Wait for the connection time out.

Additional informations

This was working perfectly fine when I was using Linux with the same client.

[DiegoGallegos4]

I concur with this. For me, it works occasionally and the only way to make it work is reinstalling docker.

I tried the steps in this issue but did not work.

In my case I try to connect to a db instance in a aws vpn. In the host I try:

telnet <db_host> <port>

and succeeds connecting to it while in the docker instance I get Host unreachable or Operation Timeout.

[YRM64]

For those developers utilizing a docker-compose file, try adding additional host properties to your Docker-compose file(s).

Try:

extra_hosts: - "internal.server.company.com: 172.162.4.1111"

This should get your Docker-machine talking to the internal network.

[cdenneen]

#2628 might be related. Work around is the following but still needs to be fixed upstream because work around is needed after reboot constantly. (https://github.com/docker/for-win/issues/1667#issuecomment-367357067)

[YRM64]

Devantoine, did you try some of the troubleshooting steps in #1803; more specifically referring to adding a UI element that allows you to customize IP address access through a range process on both edge and stable branches. Did you even see any similarities in #1803 to your use case? If so, please share the similarities/dissimilarites. Additionally, you added in your actual behavior "When I'm connecting through a VPN".........I would like to know the steps you taken to connect to the VPN. Your steps to reproduce the behavior simply states "connect to the VPN (I'm not sure which one it is...).

Diegogallegos4 indicated he'd taken the steps in #1803; I would like to see his logs, expected, and actual behavior(s), steps taken to reproduce behavior, and any other additional information he might have.

[cdenneen]

/lifecycle frozen

[mightybs]

I'm experiencing this as well on Version 18.06.1-ce-mac73 (26764) I'm not sure what version I previously had, but this was working, and now I cannot access resources on the VPN that my host is connected to. Alternatively I am able to port forward on local host to work around.

e.g. Inside container: psql -h ${vpn_ip_address} will fail, but after setting up port forwarding on my mac to ${vpn_ip_address}, e.g. ynetd -proxy ":5432 test-edgelookout-pg-20170609.cqdj8tjqocal.us-west-2.rds.amazonaws.com:5432" date that's when from inside the container: psql -h ${local_address_for_mac} does work

I've also been trying to find a working version, by installing older versions of Docker, but have not been successful in finding a working version yet.

[zhyon404]

I'm experiencing the same. and solving by change the bip setting. . docker0 using 172.17.x.x as address. so if your vpn address is in the same subnet, it will route to docker0. the above setting change the docker0 ip address. than it can rote correctly.

[mightybs]

As @zhyon404 mentioned, I ams also using the "bip" setting in the config, because I've got a network in the same default address space, despite this setting, I am unable to directly contact assets on the VPN.

But now that you've brought it up, it's got me digging further into my VPN setups to see if something looks out of whack there. I know I haven't changed VPNs or clients, but you never know. I'll do some digging and report back.

[cdenneen]

Does this only help docked or will this help the 10. addresses assigned from Kubernetes as well? On Wed, Oct 31, 2018 at 8:02 AM Brett Stauner [email protected] wrote:

As @zhyon404 https://github.com/zhyon404 mentioned, I ams also using the "bip" setting in the config, because I've got a network in the same default address space, despite this setting, I am unable to directly contact assets on the VPN.

But now that you've brought it up, it's got me digging further into my VPN setups to see if something looks out of whack there. I know I haven't changed VPNs or clients, but you never know. I'll do some digging and report back.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/docker/for-mac/issues/2820#issuecomment-434661264, or mute the thread https://github.com/notifications/unsubscribe-auth/AAr84dLd6wEjk8ms-narvjTRMVhQ_yJvks5uqZFigaJpZM4TaIiC .

[mightybs]

I just got it working again, but I'm not sure I can reproduce the failed state. At some point during my testing I did a factory reset, which still left some networking elements behind because I no longer had a "bip" entry in the daemon.json, but my containers were not using the default subnet, sadly I don't have the network inspect data from that time. Since then, I added "bip" back to the daemon.json, confirmed that the new network inspect output reflects the change, and now I can reach hosts on the VPN again!

I suspect that somewhere in all of the upgrading and troubleshooting, I had a slightly defunct network bridge. If I find that I'm having trouble again, I'll reach back out, but currently I'm back to working.

[drorata]

I believe I have the same issue even without docker-compose. I cannot access a resource from within a container but it works smoothly when accessing it directly from the host.

@zhyon404 I tried to change this bip configuration but it still didn't help. How did you decide what IP to use?

[drorata]

I used Rest to factory defaults and now it seems like I'm able to access the resources in the VPN.

[jpommerening]

@drorata thanks! I wasn't able to ping any host on the VPN network and your comment saved me from crawling (deeper) into the rabbit hole of networking :)

[cdenneen]

@drorata reset to factory defaults disables Kubernetes again. So yes you can ping hosts on VPN. When you re-enable Kubernetes it will conflict again with 10. VPN networks.

[jpommerening]

@cdenneen that's interesting. I had problems with 172. addresses though. Will confirm tomorrow whether enabling Kubernetes reintroduces the problem.

[jpommerening]

Update: Enabled k8s and can still connect to 172.x.x.x addresses.

[cdenneen]

Ok breaks with 10. VPN networks as Kubernetes uses similar. Have to modify and change to 10.x range that isn’t used by remote network. On Fri, Jan 4, 2019 at 6:13 AM Jonas [email protected] wrote:

Updates: Enabled k8s and can still connect to 172.x.x.x addresses.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/docker/for-mac/issues/2820#issuecomment-451416898, or mute the thread https://github.com/notifications/unsubscribe-auth/AAr84SRJh1h0ON5ZJfyEVCz8QMZbPbQkks5u_zdMgaJpZM4TaIiC .

[devn]

• Some suggest tunneling as a workaround
• Some say I should reset to factory defaults or reinstall.
• Others suggest using the extra_hosts stanza.
• Some say the order of connecting to the VPN and starting the docker daemon matters.
• The bip setting has been brought up.
• Finally, k8s being enabled or not has been discussed.

I do not have k8s enabled, have reset to factory defaults, tried the bip setting (tried setting to 192.168.1.5/24, and 172.17.107.0/24), tried changing the order of connecting to the VPN and starting docker. I do not have k8s enabled. My machine is at 192.168.1.8, running the OpenVPN client for mac connecting to 172.x.x.x. My linux friends don't have this issue.

I tried the extra_hosts stanza in my docker-compose file.

extra_hosts:
  - "db-host.foo.com:172.17.107.78"

On the host machine, my IP on en0 is 192.168.1.8. utun3 points at 172.27.235.171.

Any advice for troubleshooting further would be appreciated.

[devn]

While the work around is far from ideal, @mightybs solution of tunneling via ynetd worked for me. https://github.com/docker/for-mac/issues/2820#issuecomment-432528717

Also, thanks to Celmor on IRC for some additional troubleshooting help.

[DiegoGallegos4]

Just as an update and to save time debugging this issue, this issue's solution #1803; is no longer working and the docs for vpnkit are outdated.

[till]

This just happened to me today. Reset factory defaults worked — for whatever reason.

Docker version 18.09.2, build 6247962

[cdenneen]

Resetting to factory defaults disables Kubernetes. This is still issue is with Kubernetes networking conflicting with VPN networks of 10.0.0.0/8. So I’m my case I need to find a 10.x.0.0/24 to not conflict with my VPN networks at work.

[till]

Just for the record, I didn't have k8s enabled, so not sure how "related".

Also, Docker's IP range did not "conflict" with the VPN I was connected to. The only thing I enabled the other day was edge so I can --squash, but besides, my factory defaults are pretty default.

The biggest annoyance is having to log into docker hub and other custom registries now.

[tth80]

Adding another +1, had to reset docker to factory defaults. Didn't have kubernetes enabled.

Docker version 18.09.2, build 6247962

[pam-]

Another +1. The reset worked 🤷‍♀with Kubernetes enabled. Version 18.09.2, build 6247962

[kzvonov]

Had a similar VPN issue with the newest version of docker on my mac and nothing from above helped me.

The solution was to move to the previous one Hope this helps :)

[robertkern]

For what it's worth, I experience this problem on and off and doing a factory reset seems to fix it (but obviously is very annoying having to do this). I had upgraded to 2.1.0.1 hoping it might fix the issue but it didn't so downgraded to 2.0.0.2 (and did a factory reset). It's currently working (I'm currently able to access VPN network from containers going through the host computer VPN connection).

I don't customise any of the docker settings other than increasing the memory from 2GB to 4GB (so kubernetes is not enabled).

[kjenkins19]

Has anyone validated whether 2.1.0.2 fixes the issue yet, or found a more permanent solution? I try to avoid remaining behind versions.

[unstablectrl]

My VPN was also not working and I already had a different "bip" : "172.x.x.x/16" in daemon advanced options but before reseting try to delete docker networks docker network prune worked for me.

[yaizudamashii]

@kjenkins19 2.1.0.2 does not fix the issue. Doing factory reset didn't fix the problem described in the issue either. Doesn't work with 2.1.0.3 either.