moby
Add more detail on how to add use custom certificate using Rootless
Description
Context
GitLab pipeline using the docker executor with moby/buildkit:rootless.
Purpose of the job: build a Docker image and push it to an internal registry.
This registry uses a TLS certificate signed by an internal root CA, and the runner is behind a corporate proxy.
Config
In my before_script section:
- I echo my certificate in CRT format (
-----BEGIN CERTIFICATE----- ...). - Then I add:
[registry."project.url.domain"]
ca=["path/to/my/file.crt"]
to ~/.config/buildkit/buildkitd.toml.
- I also add auth credentials and proxy settings in
~/.docker/config.json.
I tried adding:
--registry-auth-tlscontext host=project.url.domain,insecure=false,ca=path/to/my/file.crt
I made sure to set no_proxy in the container environment and noProxy in Docker’s config.json so that connections to our local domain are direct.
Yet I always get:
failed to push to project.url ... TLS failed to verify x509: unknown authority
The root cert is installed and working on the host machine where the GitLab runner is running.
Attempts and workarounds
- If I add
registry.insecure=trueto:
buildctl-daemonless.sh build \
... \
--output type=image,...,push=true,registry.insecure=true
it works.
- If I add the following to
buildkit.toml:
insecure = true
it also works.
There seem to be many (~15) issues in this repo related to this situation, but with different setups: k8s, GitLab Docker executor, GitLab k8s executor, standalone setups, and of course rootless vs non-rootless.
If anyone has figured out what I’m missing here, help would be much appreciated. 🤗
Had same issue. I Just created rootless image with cat mycert.pem >> /etc/ssl/certs/ca-certificates.crt and after that removed ca=["path/to/my/file.crt"] from config and set only mirrors. Also override entrypoint with [""]. Everything works fine
So you created a custom rootless buldkit image with your cert inside /etc/.
I don't want to do this since it will imply to create a custom image for every new release.
I don't want to do this since it will imply to create a custom image for every new release.
I don't understand what other solution you are expecting. If you want to use custom ca file for buildkit inside a container, that buildkit binary needs to have some access to this file.
In buildx we have some special code to help with these transfers on docker buildx create if this is what you are looking for https://github.com/docker/buildx/blob/master/util/confutil/container.go#L27
I don't understand the difference between these two options:
[registry."project.url.domain"]
ca=["path/to/my/file.crt"]
and
--registry-auth-tlscontext host=project.url.domain,insecure=false,ca=path/to/my/file.crt
There seems to be no difference whether I use them or not — it just doesn't work. I always get : failed to push to project.url ... TLS failed to verify x509: unknown authority
My custom root CA certificate is already present in the user’s home directory, so it should be accessible by BuildKit (if I’m correct).
What I expect is that when BuildKit pushes my images to my internal registry, it uses my internal root CA certificate to verify the TLS session. However, the two options above don’t seem to have any effect, as BuildKit still appears to rely on the system certificates inside the container.
@tonistiigi is there a way (in debug logs for example) to verify which certificate buildkitD is using ?
I got this to work by writing our custom root CA to ~/certs/ca.crt and setting the env var SSL_CERT_DIR to /home/user/certs:/etc/ssl/certs, see also https://github.com/golang/go/issues/35325
I managed to push to a private repository using $CI_SERVER_TLS_CA_FILE in my gitlab-ci.yml. However, since I don’t manage the GitLab instance and $CI_SERVER_TLS_CA_FILE is a non-printable variable, I can’t determine its exact format.
Here’s the snippet I used:
- |
echo "
[registry.\"${CI_REGISTRY}\"]
ca = [\"${CI_SERVER_TLS_CA_FILE}\"]
"
I don't understand the difference between these two options:
[registry."project.url.domain"] ca=["path/to/my/file.crt"] and
--registry-auth-tlscontext host=project.url.domain,insecure=false,ca=path/to/my/file.crt
@tonistiigi any thoughts about this ?
I solved this by doing something like this:
before_script:
# setup self signed cert for buildkit
- export SSL_CERT_FILE="$HOME/my_ca_chain.pem"
- cat /etc/ssl/certs/ca-certificates.crt > "$SSL_CERT_FILE"
- echo "$MY_CRT" >> "$SSL_CERT_FILE"
I works for me, not sure why this is not mentioned anywhere
My configuration is totally valid for insecure registry setup (https://hub.docker.com/_/registry).
I am using easypki (https://github.com/google/easypki) to generate a two layer ca setup, root_ca + int_ca, and a wildcard cert for all the tools in my test lab (k8s ingress).
This comment solved my issue. https://github.com/moby/buildkit/issues/5576#issuecomment-3146296482
Self signed certificates must be trusted on buildkitd and buildctl client side together. @tonistiigi Worth for mention it the documentation.
Filed a pull request, https://github.com/moby/buildkit/pull/6228
#13 exporting to image
#13 pushing layers 2.8s done
#13 pushing manifest for harbor.tools.wkld01.pe.contoso.com/contosope/python01:dev@sha256:eae0ead01a7ab0fb478d1e0989f6b11e4cb4738945d09cf995033f22143cb99b
#13 pushing manifest for harbor.tools.wkld01.pe.contoso.com/contosope/python01:dev@sha256:eae0ead01a7ab0fb478d1e0989f6b11e4cb4738945d09cf995033f22143cb99b 0.5s done
#13 DONE 11.2s
--- Some Troubleshooting steps ---
The error message from buildkitd tells nothing more than the client side error.
#12 [stage-1 3/3] COPY ./src ./
#12 DONE 0.6s
#13 exporting to image
#13 exporting layers
#13 exporting layers 8.3s done
#13 exporting manifest sha256:9ebb41a6f1a2faa97f228feec37a581374569bccd3922163153d22c66271286c done
#13 exporting config sha256:46eabf54256b577df08e031c8e3bbe95e7dfa7697b7d809959d2e13a87929f99 done
#13 pushing layers 0.1s done
#13 ERROR: failed to push harbor.tools.wkld01.pe.contoso.com/contosope/python01:dev: failed to authorize: failed to fetch anonymous token: Get "https://harbor.tools.wkld01.pe.contoso.com/service/token?scope=repository%3Acontosope%2Fpython01%3Apull%2Cpush&service=harbor-registry": tls: failed to verify certificate: x509: certificate signed by unknown authority
------
> exporting to image:
------
error: failed to solve: failed to fetch anonymous token: Get "https://harbor.tools.wkld01.pe.contoso.com/service/token?scope=repository%3Acontosope%2Fpython01%3Apull%2Cpush&service=harbor-registry": tls: failed to verify certificate: x509: certificate signed by unknown authority
OpenSSL validation log
app@petoolbox-6fc4b6ff77-xq7rw:~/repos/contoso-pe-python-01$ openssl s_client -connect harbor.tools.wkld01.pe.contoso.com:443
CONNECTED(00000003)
depth=2 C = US, ST = WA, L = Redmond, O = Contoso, OU = PE, CN = rootca_pe
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=2 C = US, ST = WA, L = Redmond, O = Contoso, OU = PE, CN = rootca_pe
verify return:1
depth=1 C = US, ST = WA, L = Redmond, O = Contoso, OU = PE, CN = intca_pe
verify return:1
depth=0 C = US, ST = WA, L = Redmond, O = Contoso, OU = ITO, CN = *.tools.wkld01.pe.contoso.com
verify return:1
---
Certificate chain
0 s:C = US, ST = WA, L = Redmond, O = Contoso, OU = ITO, CN = *.tools.wkld01.pe.contoso.com
i:C = US, ST = WA, L = Redmond, O = Contoso, OU = PE, CN = intca_pe
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Aug 11 06:25:21 2025 GMT; NotAfter: Aug 11 06:25:21 2027 GMT
1 s:C = US, ST = WA, L = Redmond, O = Contoso, OU = PE, CN = intca_pe
i:C = US, ST = WA, L = Redmond, O = Contoso, OU = PE, CN = rootca_pe
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Aug 8 01:15:44 2025 GMT; NotAfter: Aug 6 01:15:43 2035 GMT
2 s:C = US, ST = WA, L = Redmond, O = Contoso, OU = PE, CN = rootca_pe
i:C = US, ST = WA, L = Redmond, O = Contoso, OU = PE, CN = rootca_pe
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Aug 8 01:15:37 2025 GMT; NotAfter: Aug 6 01:15:37 2035 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIID+TCCAuGgAwIBAgIQAedRpMq6E07mzOO9Ef18czANBgkqhkiG9w0BAQsFADBe
MQswCQYDVQQGEwJVUzELMAkGA1UECBMCV0ExEDAOBgNVBAcTB1JlZG1vbmQxEDAO
BgNVBAoTB0NvbnRvc28xCzAJBgNVBAsTAlBFMREwDwYDVQQDDAhpbnRjYV9wZTAe
Fw0yNTA4MTEwNjI1MjFaFw0yNzA4MTEwNjI1MjFaMHQxCzAJBgNVBAYTAlVTMQsw
CQYDVQQIEwJXQTEQMA4GA1UEBxMHUmVkbW9uZDEQMA4GA1UEChMHQ29udG9zbzEM
MAoGA1UECxMDSVRPMSYwJAYDVQQDDB0qLnRvb2xzLndrbGQwMS5wZS5jb250b3Nv
LmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKqO600/eX/Yi9iJ
L5ClYjLvsQqJCGZoHgW3a6xYWRwEsvDTEfyTjtdrVKtXC6nvo6Nxzkq01yDoKsRd
C2hpeo4OUIJiL/M7HMLhW6L42GHA5i9UFaubOrcTnCTY7S682J+xhVqwSwXOEKuA
O2qN+OQT8XB2GO7pNl8EfpunSfewJqODhPZlOqxE8R0qLJQ4UwI3nVBX2pqH+GGk
NtU7DhqjVpRVTzLde3AH3vtGeqqMmSFmRj8T40xAUXOFQphmhvp8ARd+a6QXinR/
f03pf1K0Ua0E/sgseFfHXqEgFCt0xgWqnEAr3+kOAnkJxLlc8YBr9szh5LVyp6pb
D9ZwMK8CAwEAAaOBnDCBmTAOBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYB
BQUHAwEGCCsGAQUFBwMCMB0GA1UdDgQWBBSDXpuQPtOGjK3IhUNRNzvRZ3H8IjAf
BgNVHSMEGDAWgBQfYnqk20yoqcaMtECL+aJymqjcqTAoBgNVHREEITAfgh0qLnRv
b2xzLndrbGQwMS5wZS5jb250b3NvLmNvbTANBgkqhkiG9w0BAQsFAAOCAQEAdCfb
HsqYyfbx5WAlFsPHswVcL6++gCAW/4jy/Zp76YM30KlWkQfRCPO4xpqo+ZmvDxIt
uFW2xpuCzgFYvCz5I7fZMed0L8W4Ww9vTyOrYD5e7UvVigJpa3sSzG4F14k+GHGx
CzJ0i3LjKopPaAq/mv97YBFnBp8BOCkGFaNYWzDs07UPLAmrN6npTDU7v0Q/HYAh
skCtdFvn0wFp+EvbsI8i89Ucd/VadyXPjuefvvCrknUMQGz3aMHnfCAqNYs0+lgK
oXp9SR2c6/mwUrHNpLt+XS4aUvlRKzsl6FgAEvpRm2svDwLYja0vvkFnz3bue3RU
nY6rUmps8MzbOTdjeQ==
-----END CERTIFICATE-----
subject=C = US, ST = WA, L = Redmond, O = Contoso, OU = ITO, CN = *.tools.wkld01.pe.contoso.com
issuer=C = US, ST = WA, L = Redmond, O = Contoso, OU = PE, CN = intca_pe
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3523 bytes and written 400 bytes
Verification error: self-signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self-signed certificate in certificate chain)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_128_GCM_SHA256
Session-ID: D067943E870DE06D6728CE22A76F352333E68C3B887191AA924177AECFF3DAE6
Session-ID-ctx:
Resumption PSK: A97AF062EF8A3A4307EB695BBF434D8CD5D59AEE997EC9FE1590EF4B271AD4D6
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 604800 (seconds)
TLS session ticket:
0000 - 40 60 a8 1f ea fb 3f 4b-75 c2 63 3b 66 c4 bd 49 @`....?Ku.c;f..I
0010 - 9e 2e f8 41 bb 84 e2 3f-cb 21 6f 08 60 e1 16 24 ...A...?.!o.`..$
0020 - 95 99 21 7a 9e 0f 03 ba-2f 8f 42 13 e5 96 29 bb ..!z..../.B...).
0030 - fe 54 23 40 75 5f 92 f2-b5 06 ae 6f 7e 85 ce fb .T#@u_.....o~...
0040 - e5 0b 3f 90 24 7f 15 60-f2 4c bc 97 56 f4 e9 6f ..?.$..`.L..V..o
0050 - 66 e2 94 32 f5 2b 35 5a-ff 3e 85 a0 34 f3 ac 69 f..2.+5Z.>..4..i
0060 - 34 5f ac c6 f7 36 ce 31-3d 02 00 8a 7e e3 c8 4e 4_...6.1=...~..N
0070 - 63 c
Start Time: 1758095092
Timeout : 7200 (sec)
Verify return code: 19 (self-signed certificate in certificate chain)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
HTTP/1.1 400 Bad Request
Content-Type: text/plain; charset=utf-8
Connection: close
400 Bad Requestclosed
curl - same test environment with cert installed (justify that Harbor is working)
(env) jenkins@flatcar08:~$ curl -v -u "admin:admin" -H "Content-Type: application/json" -i https://harbor.tools.wkld01.pe.contoso.com/api/v2.0/configurations
* Trying 192.168.0.68:443...
* Connected to harbor.tools.wkld01.pe.contoso.com (192.168.0.68) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
* subject: C=US; ST=WA; L=Redmond; O=Contoso; OU=ITO; CN=*.tools.wkld01.pe.contoso.com
* start date: Aug 11 06:25:21 2025 GMT
* expire date: Aug 11 06:25:21 2027 GMT
* subjectAltName: host "harbor.tools.wkld01.pe.contoso.com" matched cert's "*.tools.wkld01.pe.contoso.com"
* issuer: C=US; ST=WA; L=Redmond; O=Contoso; OU=PE; CN=intca_pe
* SSL certificate verify ok.
* using HTTP/2
* Server auth using Basic with user 'admin'
* h2h3 [:method: GET]
* h2h3 [:path: /api/v2.0/configurations]
* h2h3 [:scheme: https]
* h2h3 [:authority: harbor.tools.wkld01.pe.contoso.com]
* h2h3 [authorization: Basic YWRtaW46YWRtaW4=]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* h2h3 [content-type: application/json]
* Using Stream ID: 1 (easy handle 0x5604975437a0)
> GET /api/v2.0/configurations HTTP/2
> Host: harbor.tools.wkld01.pe.contoso.com
> authorization: Basic YWRtaW46YWRtaW4=
> user-agent: curl/7.88.1
> accept: */*
> content-type: application/json
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 200
HTTP/2 200
< content-type: application/json
content-type: application/json
< date: Wed, 17 Sep 2025 09:07:18 GMT
date: Wed, 17 Sep 2025 09:07:18 GMT
< set-cookie: sid=6b0ccc823f9cdf29907efe1bf1b379e8; Path=/; HttpOnly
set-cookie: sid=6b0ccc823f9cdf29907efe1bf1b379e8; Path=/; HttpOnly
< x-request-id: c7f4ffcf-40e8-4ac7-b3d0-f74a635b03b3
x-request-id: c7f4ffcf-40e8-4ac7-b3d0-f74a635b03b3
I don't want to do this since it will imply to create a custom image for every new release.
I don't understand what other solution you are expecting. If you want to use custom ca file for buildkit inside a container, that buildkit binary needs to have some access to this file.
In buildx we have some special code to help with these transfers on
docker buildx createif this is what you are looking for https://github.com/docker/buildx/blob/master/util/confutil/container.go#L27
The part of code may not work as expected if user separate buildctl and buildkitd into two instances, for example a vscode container with buildctl installed, buildctld is deployed to a remote builder cluster.
buildctl still requires the trust of self-signed certificates on its instance, which most of us believe the job should be done on buildkitd side, that should not be configured on buildctl instance.
Sorry I am not a GO developer that can easily pinpoint which part of code leads to the existing situation. But I can confirm other's findings, and try to include my understandings.