kwctl
kwctl copied to clipboard
kubewarden
kwctl does not honor trusted certificate authorities on the system
Is there an existing issue for this?
- [X] I have searched the existing issues
Current Behavior
kwctl pull against on-premise registry with self-signed certificate / certificate authority fails with:
kwctl pull registry://registry01.suse:5000/rke-prod/ghcr.io/kubewarden/policies/user-group-psp:v0.2.0
2022-11-22T15:54:48.099449Z WARN rustls::conn: Sending fatal alert BadCertificate
Error: the policy registry://registry01.suse:5000/rke-prod/ghcr.io/kubewarden/policies/user-group-psp:v0.2.0 could not be downloaded due to error: error sending request for url (https://registry01.suse:5000/v2/): error trying to connect: invalid peer certificate contents: invalid peer certificate: UnknownIssuer
Expected Behavior
The tool should trust the CA of the systems ca-bundle.pem.
A cli parameter to specify the trusted CA bundle should be available.
Steps To Reproduce
Use on premise registry for policies and try to use pull / push with kwctl.
Environment
- OS: SLES 15 SP4
- Architecture: x86_64
Anything else?
No response
Yes - there seems to be a way to use the sources.yaml with kwctl but here I can not specify the certificate of the CA in a way that works:
source_authorities:
"registry01.suse:5000":
- type: Data
data: |
-----BEGIN CERTIFICATE-----
MIIDhzCCAm+gAwIBAgIUVS8hv/sK6r15ZjVf/Zz/AQdoqjAwDQYJKoZIhvcNAQEL
BQAwUzELMAkGA1UEBhMCREUxGzAZBgNVBAgMEkJhZGVuLVd1ZXJ0dGVtYmVyZzEN
MAsGA1UECgwEU1VTRTEYMBYGA1UEAwwPcmVnaXN0cnkwMS5zdXNlMB4XDTIxMDMy
NDE0MTMzM1oXDTMxMDEzMTE0MTMzM1owUzELMAkGA1UEBhMCREUxGzAZBgNVBAgM
EkJhZGVuLVd1ZXJ0dGVtYmVyZzENMAsGA1UECgwEU1VTRTEYMBYGA1UEAwwPcmVn
aXN0cnkwMS5zdXNlMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAqpHI
1l8C83ON0MGkKA/s+bfJnFjBrHdeHoGbxnxqDk63cXLWIejI5QmUNGIKlGUpLnwl
It6QDtK5DRXfxR70MRDWu8p7r2yc5Vwrj/z0GqdTFFxsu+o832fukJQLAVTt2O1k
bH7PDzxsz9yewTNS5dRKmk27+kRjqIvudvUo2+LC/DmMpZiaXDXf60tgdHGhv38n
ZCqj6S2I/zh1pk2FqCz1ShQRJ8zAist/RLlO77vU30o1pHaIbEjKBj6I6O7SX6SO
NOBdDQ7zUEKmtmADVTQaDNBw1aJBpUtOYKAWlZyrwzmgXLFXPYcJLq7HEi9RdF2O
9P7SN/toZW5vsyXqxwIDAQABo1MwUTAdBgNVHQ4EFgQUlqMA2BU9MN9HRH7ebcqj
VjGxt6owHwYDVR0jBBgwFoAUlqMA2BU9MN9HRH7ebcqjVjGxt6owDwYDVR0TAQH/
BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAE9/UQ1qSi3932Lq6OJ48NkSa12V0
ezxE7fgDvm4wWK5J83yToNMp6K0uO3PfKWko7+ancgT3n2+gw5DO4/9YUplkszzo
KISVWnh+ILDHCjhzKgJNJP/zpa5bWkrvTa+WYcXqxMndYsokwd9sI/gYUtCMMngp
4VBOPlephUpUriB+sT6e/ODszXawLXNpM1oww2k0DOoxrF+KOMldVuqo/LXHOElR
amXoZIEHPRiXltNGg6NdyeSgMX0qugYyNpHRnwvwVqa9+76cOFfrQqRqq8ueeZY3
T7s2fY/AJ5Eg66wfhxSm8rNf31TWCz8eaqlTm5kd5Fzf2vwZp84pnDmLRg==
-----END CERTIFICATE-----
Results in
kwctl pull --sources-path kwctl-sources2.yaml registry://registry01.suse:5000/rke-prod/ghcr.io/kubewarden/policies/user-group-psp:v0.2.0
2022-11-22T16:23:02.422983Z WARN rustls::conn: Sending fatal alert DecodeError
Error: the policy registry://registry01.suse:5000/rke-prod/ghcr.io/kubewarden/policies/user-group-psp:v0.2.0 could not be downloaded due to error: error sending request for url (https://registry01.suse:5000/v2/): error trying to connect: invalid peer certificate encoding
Yes - there seems to be a way to use the sources.yaml with kwctl but here I can not specify the certificate of the CA in a way that works: [...] Results in
kwctl pull --sources-path kwctl-sources2.yaml registry://registry01.suse:5000/rke-prod/ghcr.io/kubewarden/policies/user-group-psp:v0.2.0 2022-11-22T16:23:02.422983Z WARN rustls::conn: Sending fatal alert DecodeError Error: the policy registry://registry01.suse:5000/rke-prod/ghcr.io/kubewarden/policies/user-group-psp:v0.2.0 could not be downloaded due to error: error sending request for url (https://registry01.suse:5000/v2/): error trying to connect: invalid peer certificate encoding
This turned out to be caused by the usage of the wildcard certificate like *.suse. Using a wildcard certificate like *.suse.local works fine.
It looks this behavior is caused by rustls, I've reached out to upstream and I talking about this issue.
The issue reported when the issue got created is valid. We have to look into that.
It looks like the reqwest crate, which is being used by oci-distribution, has a feature that enables the usage of system certificates:
rustls-tls-webpki-roots: Enables TLS functionality provided by rustls, while using root certificates from the webpki-roots crate.
This is something worth to be investigated
Opened https://github.com/kubewarden/policy-fetcher/issues/126 also related to this, they could be merged.
Working on this, will edit this post.
Patched kwctl -> policy-evaluator -> policy-fetcher -> oci-distribution with reqwests rustls-native-certs feature and rustls-native-certs as new dependencies as needed.
To test, deployed Harbor via Helm on a minikube cluster:
minikube addons enable ingress
helm upgrade -i --wait \
--namespace cert-manager \
--create-namespace \
--set installCRDs=true \
cert-manager jetstack/cert-manager
helm repo add harbor https://helm.goharbor.io
helm upgrade -i harbor harbor/harbor \
--set expose.type=ingress \
--set persistence.enabled=false \
--set expose.tls.enabled=true
setup /etc/hosts file:
core.harbor.domain
notary.harbor.domain
login at https://core.harbor.domain/ with default username & pass:
username: admin
password: Harbor12345
inspect generated certs in browser or with cert tool, check that their SAN is only
for core.harbor.domain, download and import into system cert store (don't forget to remove later). E.g:
sudo cp core-harbor-domain-chain.pem /usr/local/share/ca-certificates/core-harbor-domain-chain.crt
sudo update-ca-certificates
restart docker (minikube start again), so docker loads new certs from OS store
test harbor:
docker tag alpine core.harbor.domain/library/alpine
docker push core.harbor.domain/library/alpine
test kwctl:
kwctl push ~/.cache/kubewarden/store/registry/ghcr.io/kubewarden/policies/pod-privileged:v0.2.1 registry://core.harbor.domain/library/pod-privileged:v0.2.1