skaffold icon indicating copy to clipboard operation
skaffold copied to clipboard

Builds don't work if the docker socket isn't in `/var/run/docker.sock` location (using lima instead of Docker Desktop)

Open agxs opened this issue 3 years ago • 12 comments

Actual behavior

It looks like running skaffold run fails when the docker socket isn't in the default location of /var/run/docker.sock. Even when setting the options in the skaffold.yaml to use useDockerCLI: true the build still fails.

Expected behavior

That the docker socket file location is can be anywhere when the options in the skaffold.yaml file are set to use the docker cli and/or provide a way of overriding the expected socket location that skaffold uses.

Information

  • Skaffold version: v1.35.2
  • Operating system: MacOS 11.6.1
  • Installed via: Homebrew

I'm using limactl version 0.8.2 instead of Docker Desktop, installed via Homebrew, along with the docker cli also installed via Homebrew. VM was created using the sample docker.yaml at https://github.com/lima-vm/lima/blob/master/examples/docker.yaml

Steps to reproduce the behavior

  1. limactl start https://raw.githubusercontent.com/lima-vm/lima/master/examples/docker.yaml
  2. Run the suggested commands to configure a docker context in the message output from limactl
  3. skaffold run
  4. Receive the following error:
getting imageID for <image id-dirty>: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?. Docker build ran into internal error. Please retry.
If this keeps happening, please open an issue..

Normal docker commands work fine, just commands via skaffold have the issue.

I currently have a workaround where I manually create a symlink from /var/run/docker.sock to the socket that lima creates in $HOME/.lima/<vm>/sock/docker.sock

agxs avatar Feb 04 '22 14:02 agxs

I have the same issue. Docker context is not honored by skaffold.

814HiManny avatar Feb 09 '22 23:02 814HiManny

I'm also experiencing the same issue with colima. Would love to see Docker context support added 😁

cooperbenson-qz avatar Feb 10 '22 18:02 cooperbenson-qz

Also facing this issue trying to use Podman in place of Docker. On Mac, using a symlink to any Podman socket doesn't work either. There is ongoing discussion in the Kubernetes Slack server, channel #skaffold related to this issue.

#5739 and #5773 are both open issues that could easily provide an alternative via Minikube, but this has been deprioritized for unexplained reasons. The Minikube team appears to have done their part for this and is waiting for the Skaffold team to do the same as noted here.

Hope these breadcrumbs help others seeking a solution.

carsondwilber avatar Feb 18 '22 02:02 carsondwilber

containerd/buildkitd doesn't even have a docker socket...

afbjorklund avatar Feb 22 '22 08:02 afbjorklund

Thanks you for the issue. We would appreciate any help from the community on this issue.

tejal29 avatar Mar 22 '22 19:03 tejal29

Hi, I succeeded to run Docker build when specified DOCKER_HOST=unix:///home/dir/.colima/docker.sock

PertsevRoman avatar Apr 27 '22 09:04 PertsevRoman

i have the same issue when using docker desktop for linux in ubuntu, as the documentation says there is another context that uses another endpoint different from unix:///var/run/docker.sock, the context for docker desktop for linux is unix:///home/<user>/.docker/desktop/docker.sock

https://docs.docker.com/desktop/linux/install/#context

jearangoo avatar May 16 '22 22:05 jearangoo

I think I have the same issue, I'm trying to push an image to a registry. Docker-cli is installed but there is no Docker engine available. I thought it was the point of the useDockerCLI: true setting.

Skaffold v1.36.0

time="2022-05-31T01:01:01Z" level=warning msg="failed to list images: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?" subtask=gitlab-ci-java task=Build failed to dial gRPC: cannot connect to the Docker daemon. Is 'docker daemon' running on this host?: dial unix /var/run/docker.sock: connect: no such file or directory exit status 1. Docker build ran into internal error. Please retry. If this keeps happening, please open an issue..

Guillaume-Mayer avatar May 31 '22 01:05 Guillaume-Mayer

Hi, guys. I think I'm having a related problem. currently, we're using contained instead of docker. So we're using nerdctl as a docker-like CLI tools. After installing buildkit, now we can build images as before. But how can I let skaffold using nerdctl to build images? the error message shows that skafflod needs the docker image

I've tried several methods including create a soft link in /usr/local/sbin. But still no good. Is there any way to solve this problem? Thanks anyway.

leslieluyu avatar Jun 06 '22 09:06 leslieluyu

@agxs it looks like there are a couple places in our code where we're not respecting the user's use of useCLI. Would be great to comb through these and make sure we're using docker cli where appropriate when specified. I'm not sure our team has the bandwidth to look at this/fix this soon, but PRs are always welcome

MarlonGamez avatar Jun 14 '22 21:06 MarlonGamez

@leslieluyu would you be able to open an issue and attach logs from running with -v=debug?

MarlonGamez avatar Jun 14 '22 21:06 MarlonGamez

Similar workaround for podman users:

  1. Be running podman system service --time=0
  2. DOCKER_HOST=unix:///run/user/${UID}/podman/podman.sock skaffold run

drewp avatar Jul 11 '22 22:07 drewp

@MarlonGamez, it sounds like there's not currently a workaround for podman users not using the podman system service (and podman.sock and DOCKER_HOST) as @drewp mentioned, since there is still some code not respecting useCLI, right?

E.g. I see from our skaffold build (v1.39.2) attempting to use podman:

build [xxx.yyy.zzz/archrepo/archrepo-ui] failed: getting imageID for xxx.yyy.zzz/archrepo/archrepo-ui:versions_container_initial-4-g937a427-dirty: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/images/xxx.yyy.zzz/archrepo/archrepo-ui:versions_container_initial-4-g937a427-dirty/json": dial unix /var/run/docker.sock: connect: permission denied. Docker build ran into internal error. Please retry.
If this keeps happening, please open an issue..

jdoylei avatar Oct 04 '22 13:10 jdoylei

The way to fix this is to set DOCKER_HOST correctly or link the correct socket to the default socket path. See here for Colima and https://github.com/GoogleContainerTools/skaffold/issues/7078#issuecomment-1180979589 for podman. Opened https://github.com/GoogleContainerTools/skaffold/issues/8190 to improve skaffold's error message on this.

gsquared94 avatar Dec 05 '22 19:12 gsquared94

@gsquared94, I understand that the original issue description here was about build tools that have a docker socket but place it elsewhere, and that the DOCKER_HOST workaround applies there.

But the issue description also references "useDockerCLI: true". I think @MarlonGamez 's comment earlier mentioned that the useDockerCLI setting was a little incomplete, contributing to this problem. It sounds like build tools that do not have a docker socket at all (like using podman but without the podman system service) will still not work because of this, right?

I know there may not be bandwidth to complete the cleanup of useDockerCLI. But could an issue be created to track that it's incomplete? Right now it kind of seems that useDockerCLI doesn't do anything from the user's point of view, does it? Since skaffold is still going to use the Docker Engine API anyway via the required docker socket. And/or a documentation update clarifying the state of useDockerCLI?

Thanks for your time!

jdoylei avatar Dec 05 '22 19:12 jdoylei

We are making an effort to provide a socket for use with minikube docker-env:

  • https://github.com/kubernetes/minikube/issues/15389

Mostly since projects like skaffold have been so slow to adopt minikube image...

  • https://github.com/GoogleContainerTools/skaffold/issues/5739

The tricky part with DOCKER_HOST for ssh:, is that it also needs to be set for a non-interactive shell...

https://docs.docker.com/engine/security/rootless/#expose-docker-api-socket-through-ssh

Thus it needs to go at the very top of the ~/.bashrc (on the remote!), before it returns.

# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples

# If not running interactively, don't do anything
case $- in
    *i*) ;;
      *) return;;
esac

It also need a "docker" program to run docker system dial-stdio, but that can be fixed.

For instance you can do a small shell wrapper with socat, or some similar workaround.

afbjorklund avatar Dec 06 '22 07:12 afbjorklund