pack
pack copied to clipboard
new build flag --docker-host does not work as expected
Summary
This could be my misunderstanding of how the --docker-host
flag should be used, so let me know if the error is on my end.
I've been using the --publish
flag for the past months and have been waiting for https://github.com/buildpacks/pack/pull/988 so I could build my containers using a remote docker daemon without publishing. Now that https://github.com/buildpacks/pack/pull/988 is merged I can't seem to get the --docker-host
flag to work.
My builds exit early with:
09:59:49 ===> ANALYZING
09:59:49 ERROR: failed to get previous image: Cannot connect to the Docker daemon at tcp://127.0.0.1:2375. Is the docker daemon running?
09:59:49 ERROR: failed to build: executing lifecycle: failed with status code: 1
Reproduction
Steps
- set up a remote docker dameon
- set
DOCKER_HOST=tcp://127.0.0.1:2375
- make sure
docker info
works - download the tip of main for pack and build
- make sure
pack report
shows a commit that includes the new build flag--docker-host
- run
./pack build --publish demo/app -v --builder paketobuildpacks/builder:tiny
(The builder doesn't matter in this case since the error happens in the analyze phase trying to use docker) - notice how everything works with the
--publish
flag - run
./pack build --docker-host=inherit demo/app -v --builder paketobuildpacks/builder:tiny
(The builder doesn't matter in this case since the error happens in the analyze phase trying to use docker) - notice the
ERROR: failed to get previous image: Cannot connect to the Docker daemon at tcp://127.0.0.1:2375. Is the docker daemon running?
error in the ANALYZE phase. - run
./pack build --docker-host=tcp://127.0.0.1:2375 demo/app -v --builder paketobuildpacks/builder:tiny
(The builder doesn't matter in this case since the error happens in the analyze phase trying to use docker) - notice the
ERROR: failed to get previous image: Cannot connect to the Docker daemon at tcp://127.0.0.1:2375. Is the docker daemon running?
error in the ANALYZE phase.
Current behavior
- Builds using the
--publish
flag work - Builds not using the
--publish
flag and using the--docker-host=inherit
flag don't work - Builds not using the
--publish
flag and using the--docker-host=tcp://127.0.0.1:2375
flag don't work
Expected behavior
- Builds not using the
--publish
flag and using the--docker-host=inherit
flag do work - Builds not using the
--publish
flag and using the--docker-host=tcp://127.0.0.1:2375
flag do work
Environment
DOCKER_HOST=tcp://127.0.0.1:2375
pack info
09:59:46 Pack:
09:59:46 Version: 0.0.0+git-0c84b4a
09:59:46 OS/Arch: linux/amd64
09:59:46
09:59:46 Default Lifecycle Version: 0.10.2
09:59:46
09:59:46 Supported Platform APIs: 0.3, 0.4
09:59:46
09:59:46 Config:
09:59:46 (no config file found at /root/.pack/config.toml)
docker info
09:59:47 Client:
09:59:47 Context: default
09:59:47 Debug Mode: false
09:59:47 Plugins:
09:59:47 app: Docker App (Docker Inc., v0.9.1-beta3)
09:59:47 buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
09:59:47
09:59:47 Server:
09:59:47 Containers: 345
09:59:47 Running: 4
09:59:47 Paused: 0
09:59:47 Stopped: 341
09:59:47 Images: 2198
09:59:47 Server Version: 20.10.3
09:59:47 Storage Driver: overlay2
09:59:47 Backing Filesystem: xfs
09:59:47 Supports d_type: true
09:59:47 Native Overlay Diff: true
09:59:47 Logging Driver: json-file
09:59:47 Cgroup Driver: cgroupfs
09:59:47 Cgroup Version: 1
09:59:47 Plugins:
09:59:47 Volume: local
09:59:47 Network: bridge host ipvlan macvlan null overlay
09:59:47 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
09:59:47 Swarm: inactive
09:59:47 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
09:59:47 Default Runtime: runc
09:59:47 Init Binary: docker-init
09:59:47 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
09:59:47 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
09:59:47 init version: de40ad0
09:59:47 Security Options:
09:59:47 seccomp
09:59:47 Profile: default
09:59:47 Kernel Version: 3.10.0-1160.15.2.el7.x86_64
09:59:47 Operating System: CentOS Linux 7 (Core)
09:59:47 OSType: linux
09:59:47 Architecture: x86_64
09:59:47 CPUs: 32
09:59:47 Total Memory: 125.7GiB
09:59:47 Name: jail4-house.eng.qops.net
09:59:47 ID: 2RSL:KOJM:VI3R:VHXW:2YDN:CWLL:W6JS:HY5T:GNOW:BM5U:QBDX:E6SY
09:59:47 Docker Root Dir: /var/lib/docker
09:59:47 Debug Mode: false
09:59:47 Registry: https://index.docker.io/v1/
09:59:47 Labels:
09:59:47 Experimental: false
09:59:47 Insecure Registries:
09:59:47 127.0.0.0/8
09:59:47 Live Restore Enabled: false
09:59:47
09:59:47 WARNING: API is accessible on http://127.0.0.1:2375 without encryption.
09:59:47 Access to the remote API is equivalent to root access on the host. Refer
09:59:47 to the 'Docker daemon attack surface' section in the documentation for
09:59:47 more information: https://docs.docker.com/engine/security/security/#docker-daemon-attack-surface
09:59:47 WARNING: bridge-nf-call-iptables is disabled
09:59:47 WARNING: bridge-nf-call-ip6tables is disabled
@matejvasek (tagging you since you wrote the PR), am I using the --docker-host
flag correctly?
@wburningham when using TCP
you must ensure the address is visible from the build container. For globally visible hostnames/IP adresses this is not an issue, it's little bit more tricky with localhost
.
On Linux
you can do that by using --network=host
then host's 127.0.0.1
will be visible.
./pack build --network=host --docker-host=tcp://127.0.0.1:2375 demo/app -v --builder paketobuildpacks/builder:tiny
Another option would be pass your docker bridge address
it could look like 172.17.0.1
, you could check it via ip a
.
13: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:3b:c0:29:3e brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:3bff:fec0:293e/64 scope link
valid_lft forever preferred_lft forever
./pack build --docker-host=tcp://172.17.0.1:2375 demo/app -v --builder paketobuildpacks/builder:tiny
https://nickjanetakis.com/blog/docker-tip-65-get-your-docker-hosts-ip-address-from-in-a-container
Builds using the --publish flag work
I think I should have made --publish
and --docker-host
mutually exclusive.
wdyt @dfreilich @micahyoung ?
@matejvasek so does the --publish
flag automatically set the --network=host
option?
@matejvasek so does the --publish flag automatically set the --network=host option?
No, it doesn't.
The --publish
option is unrelated to the --network
option, it's related to the --docker-host
option. The thing is that it doesn't makes sense to use --publish
and --docker-host
together since --publish
makes build container to push docker registry (like docker.io or quay.io) not to docker daemon. If you use --publish
then --docker-host
is ignored.
I think this works as designed. The only inconvenience is that by default from build container point of view 127.0.0.1
is not the same thing as 127.0.0.1
from host point of view. What do you think @dfreilich @micahyoung ?
Builds using the --publish flag work
I guess this works because the pack
cli will upload the image to docker.io and they are downloading it.
Those upload/download may be unnecessary. @dfreilich correct me if I am wrong.
@matejvasek that lines up with my understanding - that any --docker-host=tcp://<host address>
must be independently accessible to both to pack
and the build container it creates on the daemon. Also, that adding --docker-host=tcp://127.0.0.1:2375 --network=host
should potentially provide that connectivity (at least for a Linux Docker daemon).
And, yes --publish
and --docker-host
would be mutually exclusive since the build container's DOCKER_HOST
would be ignored by lifecycle
when it writes to the registry instead of the daemon.
I think this issue does point out how some underlying complexity of pack
leaks through to the users with --docker-host
. I wonder, could a warning or maybe some connectivity validation of the host value be useful? Maybe we could start with a string check for tcp://127.0.0.*:*
or tcp://localhost:*
then warn to use --network=host
. Or maybe some other change to the UX?
(Updated for clarity)
I'm following this thread with interest as we're considering using pack in a CI/CD environment using jenkins and a remote docker daemon. Using the --docker-host parameter, I can get past the ===> ANALYZING problems, but only when using an insecure remote docker daemon.
Unfortunately, when using a docker daemon secured with --tlsverify and --tls*certs, the analyze phase now fails with a ERROR: failed to get previous image: Error response from daemon: Client sent an HTTP request to an HTTPS server.
as it seems it does not respect the DOCKER_CERT_PATH and DOCKER_TLS_VERIFY environment variables either.
@jdalbis If I understand your scenario correctly, it sounds like your use-case may already be supported with current pack, without the --docker-host
flag. You should be able to do this today:
export DOCKER_HOST=tcp://remote-host:2376
export DOCKER_CERT_PATH=/my-certs #location of your tls* certs
export DOCKER_TLS_VERIFY=1
pack build example.com/my-app:latest ... # normal params
With this setup, pack
creates its build container on the remote Docker daemon over the TLS-encrypted API. It then runs all the buildpacks there, builds your app, and depending on whether you use --publish
or not, it will write the image to a registry or to that daemon (over a locally bind-mounted socket in the container).
I put together a post from a couple months back that goes into some more depth: https://medium.com/buildpacks/pack-with-a-remote-docker-daemon-41aab804b839
But the --docker-host
flag, right now, is really only to override how that last "locally bind-mounted socket in the container" part works. It wouldn't change how pack
uses the DOCKER_*
env vars to create the container in the first place, so I'm pretty sure you may be able to do what want today.
@matejvasek This leads me to wonder if there's a mismatch between the name of the --docker-host
flag and it's implemented behavior. I wonder if it would make sense to change the flag name for now.
Maybe something like --lifecycle-docker-host
or --lifecycle-socket
but keep all the implemented behavior. That could make it easier to distinguish from pack
's DOCKER_HOST
, and probably makes it clearer what "inherit"
is applying pack
's settings to the lifecycle container.
Would you be ok with changing the name, and if so, do you have thoughts on a new name?
Perhaps (and separately from this PR) we should even add a separate flag for setting pack
's daemon host, instead of through the DOCKER_*
env vars I described in the previous comment. It could behave much like docker
-CLI's --host
, which is potentially a familiar pattern for doing this.
We should discuss this in a sub-team sync when @micahyoung is present.
Waiting on platform sub-team RFC from @micahyoung on this.
We can fix this one, by solving #1338, I am going to close it in favor of the latest one. Please reopen this ticket if someone has strong feels about it