redhat-operators-pull-secret is referenced but not created

[unixsysadmin]

Environment

OS: RHEL 8.6
Kubernetes/K3s: v1.23.6+k3s1
AWX Operator: 0.21.0

Description

The default installation on the AWX Operator makes reference to redhat-operators-pull-secret. However, this secret is never created during the deployment instructions. When running AWX on K3S an error is logged stating that the secret cannot be found and the errors continue to be logged almost every minute during lifetime. Example:

May 16 14:13:00 XXX k3s[26138]: I0516 14:13:00.442021   26138 kubelet_pods.go:891] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-675865446d-qv8xk" secret="" err="secret \"redhat-operators-pull-secret\" not found"
May 16 14:13:29 XXX k3s[26138]: I0516 14:13:29.621409   26138 kubelet_pods.go:891] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-675865446d-qv8xk" secret="" err="secret \"redhat-operators-pull-secret\" not found"
May 16 14:13:30 XXX k3s[26138]: I0516 14:13:30.624170   26138 kubelet_pods.go:891] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-675865446d-qv8xk" secret="" err="secret \"redhat-operators-pull-secret\" not found"
May 16 14:13:40 XXX k3s[26138]: I0516 14:13:40.143647   26138 kubelet_pods.go:891] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-675865446d-qv8xk" secret="" err="secret \"redhat-operators-pull-secret\" not found"

The reference to the non-existent secret should be removed.

[kurokobo]

Hi, this issue is also reproducible on my side with K3s v1.23.6+k3s1, by simply invoking make deploy to deploy Operator.

The AWX Operator works as expected anyway, even if there are many errors, but this is simply a bit annoying.

$ sudo grep redhat-operators-pull-secret /var/log/messages-20220508
May  7 14:35:22 kuro-awx01 k3s[3754253]: I0507 14:35:22.395676 3754253 kubelet_pods.go:898] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-57d56b64db-hzspb" secret="" err="secret \"redhat-operators-pull-secret\" not found"
May  7 14:35:26 kuro-awx01 k3s[3754253]: I0507 14:35:26.105450 3754253 kubelet_pods.go:898] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-57d56b64db-hzspb" secret="" err="secret \"redhat-operators-pull-secret\" not found"
May  7 14:35:27 kuro-awx01 k3s[3754253]: I0507 14:35:27.107478 3754253 kubelet_pods.go:898] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-57d56b64db-hzspb" secret="" err="secret \"redhat-operators-pull-secret\" not found"
May  7 14:35:28 kuro-awx01 k3s[3754253]: I0507 14:35:28.110076 3754253 kubelet_pods.go:898] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-57d56b64db-hzspb" secret="" err="secret \"redhat-operators-pull-secret\" not found"
May  7 14:35:30 kuro-awx01 k3s[3754253]: I0507 14:35:30.598231 3754253 kubelet_pods.go:898] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-57d56b64db-hzspb" secret="" err="secret \"redhat-operators-pull-secret\" not found"
May  7 14:36:44 kuro-awx01 k3s[3754253]: I0507 14:36:44.567874 3754253 kubelet_pods.go:898] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-57d56b64db-hzspb" secret="" err="secret \"redhat-operators-pull-secret\" not found"
May  7 14:38:06 kuro-awx01 k3s[3754253]: I0507 14:38:06.569145 3754253 kubelet_pods.go:898] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-57d56b64db-hzspb" secret="" err="secret \"redhat-operators-pull-secret\" not found"
May  7 14:39:13 kuro-awx01 k3s[3754253]: I0507 14:39:13.566469 3754253 kubelet_pods.go:898] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-57d56b64db-hzspb" secret="" err="secret \"redhat-operators-pull-secret\" not found"
May  7 14:40:40 kuro-awx01 k3s[3754253]: I0507 14:40:40.567253 3754253 kubelet_pods.go:898] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-57d56b64db-hzspb" secret="" err="secret \"redhat-operators-pull-secret\" not found"
May  7 14:42:05 kuro-awx01 k3s[3754253]: I0507 14:42:05.567008 3754253 kubelet_pods.go:898] "Unable to retrieve pull secret, the image pull may not succeed." pod="awx/awx-operator-controller-manager-57d56b64db-hzspb" secret="" err="secret \"redhat-operators-pull-secret\" not found"
...

Temprary workaround is to create dummy secret with the name redhat-operators-pull-secret in the namespace where the AWX Operator exists.

kubectl -n awx create secret docker-registry redhat-operators-pull-secret \
  --docker-server=dummy.example.com \
  --docker-username=dummy \
  --docker-password=dummy

I don't know what the redhat-operators-pull-secret is for, but I think since it is unnecessary for most of users, is it possible to make it so that users don't have to pay attention to this secret?

[kladiv]

+1

[GarisonLotus]

+1

Thanks for the temp workaround.

[aimcod]

@kurokobo thanks for the workaround. This works as expected, with a slight mention.

If the secret is already there(which it was in my case), then one has to delete the secret, then recreated it.

kubectl delete secret -n awx redhat-operators-pull-secret
kubectl -n awx create secret docker-registry redhat-operators-pull-secret   --docker-server=dummy.example.com   --docker-username=dummy   --docker-password=dummy

To note, that I have just upgraded from AWX 22.4.0/2.3.0 to 23.3.1/2.7.1.