Can't pin images if docker authentication is required.

[jrsmroz]

Bug Report

What did you do?

I've tried to pin the images using the operator-sdk. But got an error

What did you expect to see?

Images should be pinned correctly.

What did you see instead? Under which circumstances?

$ bin/kustomize build config/redhat-certified | bin/operator-sdk generate bundle --output-dir=bundle/redhat-certified --overwrite --version 0.0.1 --channels=alpha --default-channel=alpha --use-image-digests
Generating bundle version 0.0.1
Generating bundle manifests
pinning image versions to digests instead of tags
2022/09/09 10:29:11 manifests true
2022/09/09 10:29:11 skipping non-yaml file without errors: manifests
2022/09/09 10:29:11 gateway-operator-admission_rbac.authorization.k8s.io_v1_clusterrole.yaml false
2022/09/09 10:29:11 visited file or dir: "bundle/redhat-certified/manifests/gateway-operator-admission_rbac.authorization.k8s.io_v1_clusterrole.yaml"
2022/09/09 10:29:11 skipping file because it's not a ClusterServiceVersion: gateway-operator-admission_rbac.authorization.k8s.io_v1_clusterrole.yaml
2022/09/09 10:29:11 gateway-operator-admission_rbac.authorization.k8s.io_v1_clusterrolebinding.yaml false
2022/09/09 10:29:11 visited file or dir: "bundle/redhat-certified/manifests/gateway-operator-admission_rbac.authorization.k8s.io_v1_clusterrolebinding.yaml"
2022/09/09 10:29:11 skipping file because it's not a ClusterServiceVersion: gateway-operator-admission_rbac.authorization.k8s.io_v1_clusterrolebinding.yaml
2022/09/09 10:29:11 gateway-operator-admission_rbac.authorization.k8s.io_v1_role.yaml false
2022/09/09 10:29:11 visited file or dir: "bundle/redhat-certified/manifests/gateway-operator-admission_rbac.authorization.k8s.io_v1_role.yaml"
2022/09/09 10:29:11 skipping file because it's not a ClusterServiceVersion: gateway-operator-admission_rbac.authorization.k8s.io_v1_role.yaml
2022/09/09 10:29:11 gateway-operator-admission_rbac.authorization.k8s.io_v1_rolebinding.yaml false
2022/09/09 10:29:11 visited file or dir: "bundle/redhat-certified/manifests/gateway-operator-admission_rbac.authorization.k8s.io_v1_rolebinding.yaml"
2022/09/09 10:29:11 skipping file because it's not a ClusterServiceVersion: gateway-operator-admission_rbac.authorization.k8s.io_v1_rolebinding.yaml
2022/09/09 10:29:11 gateway-operator-admission_v1_serviceaccount.yaml false
2022/09/09 10:29:11 visited file or dir: "bundle/redhat-certified/manifests/gateway-operator-admission_v1_serviceaccount.yaml"
2022/09/09 10:29:11 skipping file because it's not a ClusterServiceVersion: gateway-operator-admission_v1_serviceaccount.yaml
2022/09/09 10:29:11 gateway-operator-controller-manager-metrics-service_v1_service.yaml false
2022/09/09 10:29:11 visited file or dir: "bundle/redhat-certified/manifests/gateway-operator-controller-manager-metrics-service_v1_service.yaml"
2022/09/09 10:29:11 skipping file because it's not a ClusterServiceVersion: gateway-operator-controller-manager-metrics-service_v1_service.yaml
2022/09/09 10:29:11 gateway-operator-manager-config_v1_configmap.yaml false
2022/09/09 10:29:11 visited file or dir: "bundle/redhat-certified/manifests/gateway-operator-manager-config_v1_configmap.yaml"
2022/09/09 10:29:11 skipping file because it's not a ClusterServiceVersion: gateway-operator-manager-config_v1_configmap.yaml
2022/09/09 10:29:11 gateway-operator-metrics-reader_rbac.authorization.k8s.io_v1_clusterrole.yaml false
2022/09/09 10:29:11 visited file or dir: "bundle/redhat-certified/manifests/gateway-operator-metrics-reader_rbac.authorization.k8s.io_v1_clusterrole.yaml"
2022/09/09 10:29:11 skipping file because it's not a ClusterServiceVersion: gateway-operator-metrics-reader_rbac.authorization.k8s.io_v1_clusterrole.yaml
2022/09/09 10:29:11 gateway-operator-validating-webhook_v1_service.yaml false
2022/09/09 10:29:11 visited file or dir: "bundle/redhat-certified/manifests/gateway-operator-validating-webhook_v1_service.yaml"
2022/09/09 10:29:11 skipping file because it's not a ClusterServiceVersion: gateway-operator-validating-webhook_v1_service.yaml
2022/09/09 10:29:11 gateway-operator.konghq.com_controlplanes.yaml false
2022/09/09 10:29:11 visited file or dir: "bundle/redhat-certified/manifests/gateway-operator.konghq.com_controlplanes.yaml"
2022/09/09 10:29:11 skipping file because it's not a ClusterServiceVersion: gateway-operator.konghq.com_controlplanes.yaml
2022/09/09 10:29:11 gateway-operator.konghq.com_dataplanes.yaml false
2022/09/09 10:29:11 visited file or dir: "bundle/redhat-certified/manifests/gateway-operator.konghq.com_dataplanes.yaml"
2022/09/09 10:29:11 skipping file because it's not a ClusterServiceVersion: gateway-operator.konghq.com_dataplanes.yaml
2022/09/09 10:29:11 gateway-operator.konghq.com_gatewayconfigurations.yaml false
2022/09/09 10:29:11 visited file or dir: "bundle/redhat-certified/manifests/gateway-operator.konghq.com_gatewayconfigurations.yaml"
2022/09/09 10:29:11 skipping file because it's not a ClusterServiceVersion: gateway-operator.konghq.com_gatewayconfigurations.yaml
2022/09/09 10:29:11 kong-gateway-operator.clusterserviceversion.yaml false
2022/09/09 10:29:11 visited file or dir: "bundle/redhat-certified/manifests/kong-gateway-operator.clusterserviceversion.yaml"
2022/09/09 10:29:11 Found pullspec for relatedImage kong: registry.connect.redhat.com/kong/kong:2.8.1
2022/09/09 10:29:11 Found pullspec for relatedImage kong-controller: registry.connect.redhat.com/kong/kong-ingress-controller:2.5.0-redhat
2022/09/09 10:29:11 Found pullspec for container manager: ghcr.io/kong/gateway-operator:0.0.1
2022/09/09 10:29:11 Found pullspec for container kube-rbac-proxy: gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0
2022/09/09 10:29:11 Found pullspec for kong var: registry.connect.redhat.com/kong/kong:2.8.1
2022/09/09 10:29:11 Found pullspec for kong_controller var: registry.connect.redhat.com/kong/kong-ingress-controller:2.5.0-redhat
FATA[0001] Error generating bundle manifests: error resolving image: GET https://registry.connect.redhat.com/auth/realms/rhcc/protocol/redhat-docker-v2/auth?scope=repository%3Akong%2Fkong%3Apull&service=docker-registry: UNAUTHORIZED: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication
make[1]: *** [_bundle] Error 1
make: *** [bundle.redhat-certified] Error 2

Environment

Operator type:

N/A

Kubernetes cluster type:

N/A

$ operator-sdk version

operator-sdk version: "v1.23.0", commit: "1eaeb5adb56be05fe8cc6dd70517e441696846a4", kubernetes version: "v1.24.2", go version: "go1.19", GOOS: "darwin", GOARCH: "arm64"

$ go version (if language is Go)

$ kubectl version

Possible Solution

Seems like the pinImages totally skips authentication https://github.com/operator-framework/operator-sdk/commit/dda82e56effb83f13a542bbc3151d563d50544aa#diff-9af207f9a6795d49e5653f08b2039a536ee96c56ec6043955144b2b2841e3ae3R362 The operator-manifest-tools does have a support for authentication https://github.com/operator-framework/operator-manifest-tools/blob/main/pkg/imageresolver/imageresolver.go#L82 but it's not being used in the operator-sdk

Additional context

N/A

[jtaylor-kove]

I have also encountered the same issue and when looking into it observed that the Crane library used by operator-manifest-tools will, by default, use authn.DefaultKeychain which will look for credentials in ~/.docker/config.json, $DOCKER_CONFIG, and $XDG_RUNTIME_DIR/containers/auth.json in that order. However, operator-manifest-tools specifies authn.Anonymous if no username and password are provided, and with authn.Anonymous specified Crane will not fallback to authn.DefaultKeychain and will not look for credentials. Thus, since the operator-sdk does not provide a username and password to operator-manifest tools, there is no way to provide the credentials to authenticate to a private registry when using the operator-sdk to pin image digests.

Ideally, operator-manifest-tools should also be updated to support using authn.DefaultKeychain with Crane so that credentials can be found in one of the expected file paths instead of having to provide the credentials through the operator-sdk.

[lsierant]

I can confirm this. It is a major bug as it is not possible to use image digests option for certified operator bundles which are using images from registry.connect.redhat.com.

[lsierant]

As a workaround it is possible to use operator-manifest-tools directly as an additional step. This way it is possible to use different image resolver, e.g. skopeo which supports using docker's config.json. Crane also supports it, but as @jtaylor-kove mention its usage is not correct here.

workaround:

operator-manifest-tools pinning pin -v -r skopeo -a ~/.docker/config.json <bundle-path>/manifests

[jberkhahn]

Hey there,

been working on this and have a solution to allow auth using the default keychain, but we weren't able to get it in in time, so bumping this to 1.27