docker
Remote building through SSH is hanging without explicit `--driver=docker-container` option
Description
~Building images through SSH using remote driver is not working now, since v0.9.0.
BuildX hangs after calling docker buildx build command with no stdout/stderr output.~
Remote building hangs after calling docker buildx build or docker buildx create --bootstrap until passing --driver=docker-container option explicitly, since v0.9.0.
According to @jedevc this is because BuildX waits until the server accepts connections to buildkitd instance.
I created a builder using the command line below:
docker buildx create --bootstrap --name ec2 --platform linux/arm64 ssh://siketyan@[redacted].ap-northeast-1.compute.amazonaws.com
Affected versions
v0.9.0 or newer (confirmed on both v0.9.0 and v0.9.1)
Environment
Ubuntu 20.04 LTS (amd64)
Ah, we recently changed the default driver detection logic in #1134: see https://github.com/docker/buildx/pull/1134/files#diff-577764d60d85606436fc71c2393446d6853531249d5fc9a191594feda5a1ad63R9.
Essentially, the command line you're using is attempting to connect to a raw port on your ssh target, attempting to find an instance of buildkitd listening there. Because --bootstrap is passed, the command is hanging until it can establish a connection.
To avoid the ambiguity, you can pass --driver=docker-container as a flag to force the previous behavior.
@jedevc
It worked by passing --driver=docker-container, thank you for your advice!
By the way I think this hanging behaviour is still a problem, and am going to change this issue to focus that. If you think that is a specification, not an issue, please close this issue :)
The reason it hangs is because the behavior of the remote driver when passed --bootstrap is to wait until it can successfully establish a connection. This is useful in things like CI or scripts, where you might want to explicitly wait until a builder is ready before moving onwards to the next step.
We should leave this issue open for the moment though, @tonistiigi suggested we might want to revert to the previous behavior (though I think it might invalidate #1134), so we can track that here.
Hi, @siketyan
Do you remember by any chance, did docker buildx create hang for you with the message similar to the one below?
[+] Building 20.0s (1/1) FINISHED
=> ERROR [internal] waiting for connection 20.0s
------
> [internal] waiting for connection:
------
ERROR: context deadline exceeded
I think this issue may be related to https://github.com/moby/buildkit/issues/4327 and https://github.com/moby/buildkit/issues/4268 or even be a duplicate?
@nikelborm Sorry, I do not remember anything about this issue 😢 I'll close the issue per I don't experience this anymore.
@nikelborm I think you're talking about the same experience as I did here https://github.com/moby/buildkit/issues/4268#issuecomment-1729955581
