gitpod-io
[feature-request] docker: "remote-tag" images instead of pull+tag+push in case of cache-hits
When the cache registry already contains the docker image that we want to build (cache hit) leeway currently does basically a docker pull <version-tag> && docker tag <version-tag> <new-tag> && docker push <new-tag>. This seems to spam IO on our build nodes and takes time as we do a "full" build on each branch creation/push.
It would be nice if we could just create a new manifest which contains the already existing layers instead ("remote tag").
This would be a nice feature to have indeed, particularly for saving time. So far I haven't implemented this because:
- when the Docker daemons are "warm" the pull+push basically reduces to a "remote tag" after the daemon has compared the manifest and realized all layers are already present locally and remotely. There's very little IO cost here (two requests per layer, no content transmitted) and it's almost as quick as a remote tag would be.
- the current method can be implemented without leeway calling itself, which is rather nice. So far there's no need for a special utility command branch in leeway that would be called during the build. We can perform all operations using standard tools only.
In short: I'd be surprised if the pull+push actually caused IO issues in the build nodes.
In short: I'd be surprised if the pull+push actually caused IO issues in the build nodes.
Might be that the IO spam bit far-fetched, I cannot proof. But looking at the logs of our builds (example) where each docker build takes ~10-15s (and we have a lot of those) I'd be surprised if that does not speed up using "remote tag".
We can perform all operations using standard tools only.
That's a fair point. This post describes a method which uses curl to do that (we'd still need to get credentials from somewhere).
Might be that the IO spam bit far-fetched, I cannot proof. But looking at the logs of our builds (example) where each docker build takes ~10-15s (and we have a lot of those) I'd be surprised if that does not speed up using "remote tag".
I'd reckon the remote tag would reduce this operation to less than 1 second (10x improvement). Because all those "push/pull" ops run at the same though, the total cost in terms of build time are low and compared to the proper build on the same order of magnitude.
But yes, there would be a ~10sec improvement with remote tag.
We can perform all operations using standard tools only.
That's a fair point. This post describes a method which uses
curlto do that (we'd still need to get credentials from somewhere).
The remote pull/push is easy enough to implement in Go because of the docker distribution/containerd libraries. It would impose a bit of a dependency burden though and make the leeway binary bigger.
The curl solution would not come with the dependency cost, but feels hackish and somewhat brittle.
But yes, there would be a ~10sec improvement with remote tag.
I have the strong feeling that the number of parallel tasks slows down the build, either directly (CPU) or indirectly (I/O bandwith, which is tightly coupled to CPU on our rather small cloud VMs). But again I cannot prove. I guess I'll hack sth to see if there are any performance gains. :+1:
I have the strong feeling that the number of parallel tasks slows down the build, either directly (CPU) or indirectly (I/O bandwith, which is tightly coupled to CPU on our rather small cloud VMs)
Totally agree. We could introduce an upper limit for parallel tasks. Leeway already has a build lock mechanism, that could be extended to limit the number of concurrent build steps.