buildkit 401 when using port in registry url

Contributing guidelines and issue reporting guide

[x] I've read the contributing guidelines and wholeheartedly agree. I've also read the issue reporting guide.

Well-formed report checklist

[x] I have found a bug that the documentation does not mention anything about my problem
[x] I have found a bug that there are no open or closed issues that are related to my problem
[x] I have provided version/information about my environment and done my best to provide a reproducer

Description of bug

Context

We have different organizations in Quay that have different tokens to be accessed. Until now, it isn't possible to separate auths by path, so with docker, we did something like this:

config.json

{
  "auths": {
     "my-registry.company:443/org1": {
      "auth": "<token1>",
      "email": ""
    },
    "my-registry.company/org2": {
      "auth": "<token2>",
      "email": ""
    }
    }
  }
}

This is mostly used in CI systems on Gitlab and it allows to pull a base image from org2 and push a built image to org1 in the same go.

Bug description

With buildkit (docker-container driver), the above doesn't work and the push to my-registry.company:443/org1 fails with 401.

docker buildx build . -t my-registry.company:443/org1/my-image:test --push --platform linux/arm64,linux/amd
64
[+] Building 1.9s (25/25) FINISHED
[...]
 => [auth] org1/my-image:test:pull,push token for my-registry.company:443                                                                                 0.0s
------
 > exporting to image:
------
ERROR: failed to build: failed to solve: failed to push my-registry.company:443/org1/my-image:test: unauthorized: access to the requested resource is not authorized

buildkit logs:

time="2025-09-10T08:56:10Z" level=debug msg="do request" digest="sha256:5664b15f108bf9436ce3312090a767300800edbbfd4511aa1a6d64357024d5dd" mediatype=application/vnd.oci.image.layer.v1.tar+gzip request.header.accept="application/vnd.oci.image.layer.v1.tar+gzip, */*" request.header.user-agent=buildkit/v0.23 request.method=HEAD size=168 span="exporting to image" spanID=f845ea6fb73917fe traceID=eb861e1cf4d306150a82544b6b05b2ff url="https://my-registry.company:443/v2/org1/my-image/blobs/sha256:5664b15f108bf9436ce3312090a767300800edbbfd4511aa1a6d64357024d5dd"
time="2025-09-10T08:56:11Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = failed to push my-registry.company:443/org1/my-image:test: unauthorized: access to the requested resource is not authorized" spanID=694f67656c70a915 traceID=eb861e1cf4d306150a82544b6b05b2ff
failed to push my-registry.company:443/org1/my-image: unauthorized: access to the requested resource is not authorized
7 v0.23.2 buildkitd --config /etc/buildkit/buildkitd.toml --allow-insecure-entitlement=network.host
github.com/moby/buildkit/exporter/containerimage.(*imageExporterInstance).Export

This is very weird as it seems to pick the correct token/scope but it still doesn't work.

Reproduction

The simplest way to reproduce this is like this:

Set up config.json for the same registry, once with a port and once without the port, using the same token.

{
  "auths": {
     "registry.company:443": {
      "auth": "<token>",
      "email": ""
    },
     "registry.company": {
      "auth": "<token>",
      "email": ""
    },
  }
}

Run docker buildx for both registry entries.

docker buildx build -t registry.company/test --push --platform linux/arm64,linux/amd64 docker buildx build -t registry.company:443/test --push --platform linux/arm64,linux/amd64

The first one works, the second one will fail even though using the same token.

Version information

Docker version 28.3.3, build 980b856 buildkit/v0.23

I'm happy to provide more info but tried to keep it concise.

Sep 10 '25 10:09 hghtwr

This is mostly used in CI systems on Gitlab

Can you check if this is similar to https://github.com/moby/buildkit/issues/6187#issuecomment-3257404509 ?

Sep 10 '25 12:09 crazy-max

@crazy-max Thank you for your quick response.

I currently fail to see how it is related as the described behavior happens on Gitlab Runners, but also on my local machine. We are not using docker login at all at any time but completly rely on the config.json with read-write tokens.

When looking at the minimal reproduction example, the error even occurs using the same registry and same service account/token with the only difference being the url called.

I think it's an explicit problem as the same configuration works perfectly fine when using "traditional" docker push or when using buildx with the (legacy) docker driver.

Sep 10 '25 15:09 hghtwr

so with docker, we did something like this:

I don't see how this could work. The key in the auth config is hostname. To support multiple credentials for same hostname would require changes in docker/cli config format.

The port doesn't work as :443 or :80 default port is defined by the insecure-registry configuration (there is a fallback from tls registry to http if registry is marked both insecure and http), so you can't have both tls and non-tls registry at same host and connect to both separately.

Sep 11 '25 00:09 tonistiigi

I'm not sure if I got your comment correctly. We are not using insecure or plain http. Both registries are called using TLS.

We just separate different keys in the docker config for the same host by adding "443" to one of them because it cannot be separated by path. Don't get me wrong, we know we are exploiting some kind of hacky behavior here, where the port is used to separate hosts but doesn't interfere with the configuration but that's all we got to solve this problem.

It very clearly works to separate the usage of different tokens for the same registry (for different orgs within the registry) when not using buildkit:

{
  "auths": {
     "registry.company:443": {
      "auth": "<TOKEN FOR ORG 2>",
      "email": ""
    },
     "registry.company": {
      "auth": "<TOKEN FOR ORG 1>",
      "email": ""
    },
  }
}

docker push my-registry.company:443/org1/test:test  ## Trying to push with token for org 2 because we use host "my-registry.company:443"
The push refers to repository [my-registry.company:443/org1/test]
418dccb7d85a: Preparing 
unauthorized: access to the requested resource is not authorized
---
docker push my-registry.company/org1/test:test ## Now using the token for org1 
The push refers to repository [my-registry.company/org1/test]
418dccb7d85a: Pushed 
test: digest: sha256:7b9b6a044d921dfcaea2a843ff19d725948590352198f93cb878fd2c19d7ba3c

The other way around:

docker push my-registry.company/org2/test:test ## Pushing to org2 with the token of org1 
The push refers to repository [my-registry.company/org2/test]
418dccb7d85a: Preparing 
unauthorized: access to the requested resource is not authorized
---
docker push my-registry.company:443/org2/test:test ## Pushing to org 2 with correct token of host "my-registry.company:443"
The push refers to repository [my-registry.company:443/org2/test]
418dccb7d85a: Pushed 
test: digest: sha256:7b9b6a044d921dfcaea2a843ff19d725948590352198f93cb878fd2c19d7ba3c size: 527

But, when using buildkit, it fails.

I'm interested if there's a specific reason why it fails with buildkit or if maybe there's even a better way to separate configs by registry paths with buildkit (as this is the original problem we want to solve).

Sep 11 '25 08:09 hghtwr

registry.company:443 and registry.company are not separate hostnames as 443 is the default port. If something does not see them as equal, then it is some kind of unexpected side-effect.

Sep 11 '25 17:09 tonistiigi

@tonistiigi So, do I understand your answer correctly, that even though, this works in the docker driver, it is neither planned nor designed to ever work when using buildkit?

Don't get me wrong, I understand if that's the case. I just want to understand what are the options here.

I'm trying to understand how Buildkit matches the hostnames and tokens, better. Can you maybe point me to the relevant code parts? I can see that the host is matched including the port number but if you say it doesn't make a difference than that might also explain why the authentication doesn't work.

 => [auth] org1/my-image:test:pull,push token for my-registry.company:443                                                                                 0.0s

Sep 12 '25 05:09 hghtwr

registry.company:443 and registry.company are not separate hostnames as 443 is the default port. If something does not see them as equal, then it is some kind of unexpected side-effect.

I think that amiguity has been there since "forever";

https://github.com/distribution/distribution/issues/2511
https://github.com/moby/moby/issues/36263

Credentials are stored as hostname[:<port>], which is used in situations where different services are running on different ports (e.g. a registry on localhost:5000, a different registry on localhost:5001, and a webserver on localhost:80); this also means that :<port> is used to resolve the matching credentials.

The host[:<port>] to use is deducted from the image reference (e.g. docker pull localhost:5001/my/image will pick credentials for localhost:5001), which means that a docker pull localhost:443/my-image will lookup credentials using localhost:443, not normalizing to localhost.

I need to dig deep (very deep) in memory about not normalizing those, but ISTR there was some situation where HSTS (HTTP Strict Transport Security) could not be applied and including :443 had to be used to prevent credentials being sent over plain text to some unrelated (shared hosting? proxy on localhost?) service on port 80. I have some vague recollection that I once did apply the normalization (as the "insecure-registries" config should be leading for this), but that it broke some cases where it was used as a loophole (such as this ticket).

Sep 12 '25 07:09 thaJeztah

@thaJeztah Thank you for these insights.

I understand that there was a very specific problem that introduced this behavior and that you removed it in favor of solving the problem "the right way", thus now normalizing the hostnames.

Our original problem is that we have different tokens for different organizations in the same registry (separated by path). Do you happen to know if there's any other possibility to sort these out instead of relying on this loophole?

Sep 12 '25 09:09 hghtwr