for-win icon indicating copy to clipboard operation
for-win copied to clipboard

Published 20 hours ago • verified publisher icon docker

Containers can't connect to the internet or to each other via host ip

Open Orski174 opened this issue 1 year ago • 3 comments

Description

ive been using docker for well over 3 years, nothing too complicated, just running some containers for my media server on docker desktop, fast forward till a couple weeks ago where i started noticing my containers arent reaching the internet nor able to communicate with each other using host ip (I have a stack for each service so i'm not using interstack communication) running a docker diagnose check in that state will only return it's unable to reach docker.net for hyper-v and on wsl. So i'm starting to think my vpnkit driver might be corrupted because after exiting docker desktop completely then restarting it works for a bit, could range from few hours to a couple days. I should also note that this issue first started when my soulseek client slskd failed in some way and killed connectivity to docker and resulted in the same thing but with slskd in docker a restart wouldnt fix it when i deleted it it seemed to work until as i said before it just dies

Reproduce

  1. Run docker
  2. Wait a few hours (usually overnight)
  3. Internet connection lost and containers can't connect to each other (container A bound to host:A container B bound to host:B trying to ping from container A to B by pinging host_ip:B fails)

Expected behavior

No response

docker version

Client:
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:08:44 2023
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.28.0 (139021)
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:14:25 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    24.0.7
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1-desktop.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.6-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.24
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.22
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.0.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.5.0
Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 18
  Running: 18
  Paused: 0
  Stopped: 0
 Images: 16
 Server Version: 25.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.146.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 2.908GiB
 Name: docker-desktop
 ID: ef6d31cc-ef9d-4d9b-9a1c-d259fe3b6450
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

Diagnostics ID

Diagnostics were not uploaded. Reason: "{"message":"[2024-03-28T14:28:49.164014200Z][com.docker.diagnose.exe.settingsloader][I] settings get: {\"acceptCanaryUpdates\":false,\"activeOrganizationName\":\"\",\"allowBetaFeatures\":true,\"allowExperimentalFeatures\":true,\"analyticsEnabled\":true,\"autoDownloadUpdates\":false,\"autoPauseTimedActivitySeconds\":30,\"autoPauseTimeoutSeconds\":300,\"autoStart\":true,\"backupData\":false,\"blockDockerLoad\":false,\"containerTerminal\":\"integrated\",\"cpus\":6,\"credentialHelper\":\"docker-credential-wincred.exe\",\"customWslDistroDir\":\"\",\"dataFolder\":\"C:\\\\ProgramData\\\\DockerDesktop\\\\vm-data\",\"defaultSnapshotter\":\"overlayfs\",\"deprecatedCgroupv1\":false,\"disableHardwareAcceleration\":false,\"disableUpdate\":false,\"diskFlush\":\"os\",\"diskSizeMiB\":65536,\"diskStats\":\"\",\"diskTRIM\":true,\"displayRestartDialog\":true,\"displaySwitchWinLinContainers\":false,\"displayed18362Deprecation\":false,\"displayedElectronPopup\":[],\"displayedOnboarding\":true,\"dockerAppLaunchPath\":\"\",\"dockerBinInstallPath\":\"system\",\"dockerDebugDefaultEnabled\":false,\"dogfoodFeatureFlagsEnabled\":false,\"eciDockerSocketCmdList\":\"[]interface {}\",\"eciDockerSocketCmdListType\":\"...\",\"eciDockerSocketImgList\":\"[]interface {}\",\"enableDefaultDockerSocket\":true,\"enableIntegrationWithDefaultWslDistro\":true,\"enableIntegrityCheck\":true,\"enableSegmentDebug\":false,\"enableSocksProxyPort\":false,\"enableWasmShims\":false,\"enhancedContainerIsolation\":false,\"exposeDockerAPIOnTCP2375\":false,\"extensionsEnabled\":true,\"extensionsPrivateMarketplace\":false,\"extensionsPrivateMarketplaceAdminContactURL\":\"\",\"filesharingDirectories\":[\"C:\\\\Tools\",\"O:\\\\\"],\"integratedWslDistros\":[],\"kernelForUDP\":false,\"kubernetesEnabled\":false,\"kubernetesImagesRepository\":\"\",\"kubernetesInitialInstallPerformed\":false,\"lastLoginDate\":0,\"latestBannerKey\":\"\",\"licenseTermsVersion\":2,\"lifecycleTimeoutSeconds\":600,\"memoryMiB\":12288,\"networkType\":\"gvisor\",\"noWindowsContainers\":false,\"onlyMarketplaceExtensions\":false,\"openUIOnStartupDisabled\":false,\"overrideProxyExclude\":\"\",\"overrideProxyHttp\":\"\",\"overrideProxyHttps\":\"\",\"overrideProxyPAC\":\"\",\"overrideWindowsDockerdPort\":\"float64\",\"proxyHttpMode\":\"s...m\",\"requireVmnetd\":true,\"runWinServiceInWslMode\":false,\"sbomIndexing\":true,\"settingsVersion\":35,\"showAnnouncementNotifications\":true,\"showExtensionsSystemContainers\":false,\"showGeneralNotifications\":true,\"showInstallScreen\":true,\"showKubernetesSystemContainers\":false,\"showPromotionalNotifications\":true,\"showSurveyNotifications\":true,\"skipUpdateToWSLPrompt\":true,\"skipWSLMountPerfWarning\":false,\"socksProxyPort\":0,\"swapMiB\":1024,\"synchronizedDirectories\":[],\"themeSource\":\"system\",\"updateAvailableTime\":0,\"updateHostsFile\":false,\"updateInstallTime\":0,\"useBackgroundIndexing\":true,\"useContainerdSnapshotter\":false,\"useCredentialHelper\":true,\"useGrpcfuse\":true,\"useNightlyBuildUpdates\":false,\"useResourceSaver\":true,\"useVirtualizationFramework\":false,\"useVirtualizationFrameworkRosetta\":false,\"useVirtualizationFrameworkVirtioFS\":false,\"useVpnkit\":true,\"useWindowsContainers\":false,\"vpnKitAllowedBindAddresses\":\"0.0.0.0\",\"vpnKitMTU\":1500,\"vpnKitMaxConnections\":2000,\"vpnKitMaxPortIdleTime\":\"float64\",\"vpnKitTransparentProxy\":true,\"vpnkitCIDR\":\"1...4\",\"wslEnableGrpcfuse\":false,\"wslEngineEnabled\":true,\"wslInstallMode\":\"installLatestWsl\",\"wslUpdateRequired\":false}\n[2024-03-28T14:28:49.168535400Z][com.docker.diagnose.exe.httpproxy][I] SetSettings: manual override disabled\n[2024-03-28T14:28:49.172808100Z][com.docker.diagnose.exe.httpproxy][I] SetSystemProxies: per-URL proxy\n[2024-03-28T14:28:49.172808100Z][com.docker.diagnose.exe.httpproxy][I] SetAdminSettings: admin settings disabled\n[2024-03-28T14:28:49.173326000Z][com.docker.diagnose.exe.public][I] using HTTP proxy inside com.docker.backend\nError uploading: uploading file \"C:\\\\Users\\\\Orski\\\\AppData\\\\Local\\\\Temp\\\\25043798-9717-4678-BD89-39EF01E6BAAE\\\\20240328142741.zip\" with key \"incoming/3/25043798-9717-4678-BD89-39EF01E6BAAE/20240328142741.zip\": RequestError: send request failed\ncaused by: Put \"https://docker-pinata-support.s3.amazonaws.com/incoming/3/25043798-9717-4678-BD89-39EF01E6BAAE/20240328142741.zip\": context deadline exceeded\n[2024-03-28T14:32:49.836222400Z][com.docker.diagnose.exe][I] If your computer needs an HTTPS proxy, please set the HTTPS_PROXY environment variable and try again.\n: exit status 1"}\n{"message":"[2024-03-28T14:28:49.164014200Z][com.docker.diagnose.exe.settingsloader][I] settings get: {\"acceptCanaryUpdates\":false,\"activeOrganizationName\":\"\",\"allowBetaFeatures\":true,\"allowExperimentalFeatures\":true,\"analyticsEnabled\":true,\"autoDownloadUpdates\":false,\"autoPauseTimedActivitySeconds\":30,\"autoPauseTimeoutSeconds\":300,\"autoStart\":true,\"backupData\":false,\"blockDockerLoad\":false,\"containerTerminal\":\"integrated\",\"cpus\":6,\"credentialHelper\":\"docker-credential-wincred.exe\",\"customWslDistroDir\":\"\",\"dataFolder\":\"C:\\\\ProgramData\\\\DockerDesktop\\\\vm-data\",\"defaultSnapshotter\":\"overlayfs\",\"deprecatedCgroupv1\":false,\"disableHardwareAcceleration\":false,\"disableUpdate\":false,\"diskFlush\":\"os\",\"diskSizeMiB\":65536,\"diskStats\":\"\",\"diskTRIM\":true,\"displayRestartDialog\":true,\"displaySwitchWinLinContainers\":false,\"displayed18362Deprecation\":false,\"displayedElectronPopup\":[],\"displayedOnboarding\":true,\"dockerAppLaunchPath\":\"\",\"dockerBinInstallPath\":\"system\",\"dockerDebugDefaultEnabled\":false,\"dogfoodFeatureFlagsEnabled\":false,\"eciDockerSocketCmdList\":\"[]interface {}\",\"eciDockerSocketCmdListType\":\"...\",\"eciDockerSocketImgList\":\"[]interface {}\",\"enableDefaultDockerSocket\":true,\"enableIntegrationWithDefaultWslDistro\":true,\"enableIntegrityCheck\":true,\"enableSegmentDebug\":false,\"enableSocksProxyPort\":false,\"enableWasmShims\":false,\"enhancedContainerIsolation\":false,\"exposeDockerAPIOnTCP2375\":false,\"extensionsEnabled\":true,\"extensionsPrivateMarketplace\":false,\"extensionsPrivateMarketplaceAdminContactURL\":\"\",\"filesharingDirectories\":[\"C:\\\\Tools\",\"O:\\\\\"],\"integratedWslDistros\":[],\"kernelForUDP\":false,\"kubernetesEnabled\":false,\"kubernetesImagesRepository\":\"\",\"kubernetesInitialInstallPerformed\":false,\"lastLoginDate\":0,\"latestBannerKey\":\"\",\"licenseTermsVersion\":2,\"lifecycleTimeoutSeconds\":600,\"memoryMiB\":12288,\"networkType\":\"gvisor\",\"noWindowsContainers\":false,\"onlyMarketplaceExtensions\":false,\"openUIOnStartupDisabled\":false,\"overrideProxyExclude\":\"\",\"overrideProxyHttp\":\"\",\"overrideProxyHttps\":\"\",\"overrideProxyPAC\":\"\",\"overrideWindowsDockerdPort\":\"float64\",\"proxyHttpMode\":\"s...m\",\"requireVmnetd\":true,\"runWinServiceInWslMode\":false,\"sbomIndexing\":true,\"settingsVersion\":35,\"showAnnouncementNotifications\":true,\"showExtensionsSystemContainers\":false,\"showGeneralNotifications\":true,\"showInstallScreen\":true,\"showKubernetesSystemContainers\":false,\"showPromotionalNotifications\":true,\"showSurveyNotifications\":true,\"skipUpdateToWSLPrompt\":true,\"skipWSLMountPerfWarning\":false,\"socksProxyPort\":0,\"swapMiB\":1024,\"synchronizedDirectories\":[],\"themeSource\":\"system\",\"updateAvailableTime\":0,\"updateHostsFile\":false,\"updateInstallTime\":0,\"useBackgroundIndexing\":true,\"useContainerdSnapshotter\":false,\"useCredentialHelper\":true,\"useGrpcfuse\":true,\"useNightlyBuildUpdates\":false,\"useResourceSaver\":true,\"useVirtualizationFramework\":false,\"useVirtualizationFrameworkRosetta\":false,\"useVirtualizationFrameworkVirtioFS\":false,\"useVpnkit\":true,\"useWindowsContainers\":false,\"vpnKitAllowedBindAddresses\":\"0.0.0.0\",\"vpnKitMTU\":1500,\"vpnKitMaxConnections\":2000,\"vpnKitMaxPortIdleTime\":\"float64\",\"vpnKitTransparentProxy\":true,\"vpnkitCIDR\":\"1...4\",\"wslEnableGrpcfuse\":false,\"wslEngineEnabled\":true,\"wslInstallMode\":\"installLatestWsl\",\"wslUpdateRequired\":false}\n[2024-03-28T14:28:49.168535400Z][com.docker.diagnose.exe.httpproxy][I] SetSettings: manual override disabled\n[2024-03-28T14:28:49.172808100Z][com.docker.diagnose.exe.httpproxy][I] SetSystemProxies: per-URL proxy\n[2024-03-28T14:28:49.172808100Z][com.docker.diagnose.exe.httpproxy][I] SetAdminSettings: admin settings disabled\n[2024-03-28T14:28:49.173326000Z][com.docker.diagnose.exe.public][I] using HTTP proxy inside com.docker.backend\nError uploading: uploading file \"C:\\\\Users\\\\Orski\\\\AppData\\\\Local\\\\Temp\\\\25043798-9717-4678-BD89-39EF01E6BAAE\\\\20240328142741.zip\" with key \"incoming/3/25043798-9717-4678-BD89-39EF01E6BAAE/20240328142741.zip\": RequestError: send request failed\ncaused by: Put \"https://docker-pinata-support.s3.amazonaws.com/incoming/3/25043798-9717-4678-BD89-39EF01E6BAAE/20240328142741.zip\": context deadline exceeded\n[2024-03-28T14:32:49.836222400Z][com.docker.diagnose.exe][I] If your computer needs an HTTPS proxy, please set the HTTPS_PROXY environment variable and try again.\n: exit status 1"}\n"

Additional Info

No response

Orski174 avatar Mar 28 '24 14:03 Orski174

I too having a similar issue which has cost me a lot of time and sleep. I'm using a docker bridge network and when the problem occurs, containers within the network can communicate with each other but cannot reach the outside word via http. DNS + PING do work and incoming HTTP continues to work, as do websocket connections made from outside.

image

Have tried many things such as making sure times are synced, uninstalling Hyper-V, scouring diagnostics logs, decrease nginx maximum proxy read and send timeouts, recording times the issue resurfaces. No real pattern found. The last three times have been:

  • 5/4: 12:04AM
  • 5/4: 2:00PM
  • 5/4: 9:45PM

I think I'll be forced to move the containers to a native Linuxhost as I can't keep dealing with this problem. Only restarting docker desktop engine re-enables http outbound traffic which obviously restarts all my containers. Seemed to start when I upgraded docker desktop a few months ago to v4.28.0.

bendruitt avatar Apr 05 '24 11:04 bendruitt

It's really annoying, seeing that everything is working but then randomly it just stops. I also will move to a linux host, but I do hope if you could find a fix so that I can keep deploying containers on my windows machine.

Orski174 avatar Apr 05 '24 12:04 Orski174

I had this issue with 4.28 and had to rollback to 4.27.2 to get it working stable again. It could be working for a few minutes to a couple hours then my containers would just suddenly lose connection and I'd get 502 errors, even tho internet connectivity was never lost.

I updated to 4.29 and haven't had the issue again. I've also made sure my WSL2 was updated to 2.1.5 as I was on 2.0.14 for a while.

Ryderjj89 avatar Apr 10 '24 16:04 Ryderjj89