Add --config option for "docker buildx create"? To better support insecure registries.

[dotnofoolin]

I'm running my own private docker registry, insecurely, on a private internal IP address. After some dockerd config changes (setting insecure-registry, etc), I can login to my registry just fine from my builder server and from my destination server.

Now when I run kamal setup, when the build step finishes, I receive an error that basically says it's trying to hit my registry server via HTTPS when I'm only running it via HTTP.

ERROR: failed to solve: failed to push 10.254.252.178/tbtregistry/hunter:aeeba309129a0634ac9ab5360d4d51fe91093ac3: failed to do request: Head "https://10.254.252.178/v2/tbtregistry/hunter/blobs/sha256:b014df6b9adf4a6766b34822ce1da17d587e5bd6c93ae5ee469842f152338fa3": dial tcp 10.254.252.178:443: connect: connection refused

Some research lead me to figure out that the buildx/build system doesn't use the same config as the regular dockerd does, so the insecure-registries is ignored when a docker buildx build --push is done.

The builder has its own config file, and a custom one can be specified during the docker buildx create command with --config.

I've monkey-patched my local kamal gem to look like:

# kamal-1.8.3/lib/kamal/commands/builder/multiarch.rb
class Kamal::Commands::Builder::Multiarch < Kamal::Commands::Builder::Base
  def create
    docker :buildx, :create, "--config /etc/buildkit/buildkitd.toml", "--use", "--name", builder_name
  end
end

And added a config file:

$ cat /etc/buildkit/buildkitd.toml
insecure-entitlements = [ "network.host", "security.insecure"]

[registry."10.254.252.178"]
  http = true
  insecure = true

And now my kamal setup works as expected.

Should there be a config key in the builder section of deploy.yml to specify this file?

[djmb]

In Kamal 2, you can tell docker to use the docker driver which should pick up the local settings. The driver is limited though - you can't build multiarch images, remote builders or build caches.

Something like this might work (assuming you want an amd64 image):

builder:
  arch: amd64
  driver: docker

[tisba]

Being able to configure the buildx instance is certainly an advanced use case, but not a very uncommon one. This should work both for local and remote builds alike. There are a ton of options that are relevant, like setting mirrors, using custom CAs, debug logging, … (see https://docs.docker.com/build/buildkit/toml-configuration/).

[iaroslavb]

Using Kamal with self-hosted Docker registries with self-signed CAs is indeed a useful feature to have.

Currently Kamal with docker-container default driver refuses to push images to Docker registries with self-signed certificates. When switching to docker driver for buildx push, as @djmb pointed out, it works just fine.

[iaroslavb]

Alternative workaround (without patching kamal) would be to (re)-create buildx builder manually, using naming convention used by kamal, but with proper command line, which will include --config=/etc/buildkitd.toml option. As of kamal 2.3.0 naming convention for local buildx builders is:

 "kamal-local-#{driver}"

where driver is docker-container.

Resulting command would be:

docker buildx create --name kamal-local-docker-container --driver=docker-container --config=/etc/buildkitd.toml

As such builder complies with kamal naming expectation, it will be used AS IS to spawn builder container with desired configuration.

Buildkit config file can be adapted to use either unsafe HTTP docker registry or registry with self-signed certificate. In my case - registry with self-signed cert - i have to put CA cert file into, for example, /etc/certs/myregistry.pem and use following buildkitd.toml (reference here )

[registry."registry:5000"]
  ca=["/etc/certs/myregistry.pem"]

You can also reuse existing CA cert for registry, configured on docker daemon level

[registry."registry:5000"]
  ca=["/etc/docker/certs.d/registry:5000/ca.crt"]

[optimuspwnius]

I'm running an insecure registry on a VM so I added this to my local docker config: "insecure-registries":["1.2.3.4"]. Replace with your registry's ip address. Then I forked kamal and appended ,registry.insecure=true to output=type as shown below.

https://github.com/basecamp/kamal/blob/main/lib/kamal/commands/builder/base.rb.

def push(export_action = "registry", tag_as_dirty: false)
  docker :buildx, :build,
  "--output=type=#{export_action},registry.insecure=true", # <--- 🎉👍
  ...
end