operator-lifecycle-manager
operator-lifecycle-manager copied to clipboard
Published
20 hours ago •
operator-framework
operator-framework
Package server APIServer unreachable after install on EKS with Istio
Bug Report
What did you do? A clear and concise description of the steps you took (or insert a code snippet).
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.19.1/crds.yaml
- https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.19.1/olm.yaml
kubectl apply -k .
What did you expect to see? A clear and concise description of what you expected to happen (or insert a code snippet).
OLM Services come up and I am able to install an Operator from OperatorHub.io
What did you see instead? Under which circumstances? A clear and concise description of what you expected to happen (or insert a code snippet).
OLM Services came up according to this.
kg pod,svc -n olm
NAME READY STATUS RESTARTS AGE
pod/catalog-operator-765c45774c-dsvm9 2/2 Running 1 166m
pod/olm-operator-9bb87877b-zmr7f 2/2 Running 1 166m
pod/operatorhubio-catalog-fq7qn 2/2 Running 0 61m
pod/packageserver-568d867fff-bzsjb 2/2 Running 0 13m
pod/packageserver-568d867fff-t7hs8 2/2 Running 0 62m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/operatorhubio-catalog ClusterIP 10.100.177.142 <none> 50051/TCP 166m
service/packageserver-service ClusterIP 10.100.127.51 <none> 5443/TCP 2m22s
packserver csv stuck in Installing State
kg csv -n olm
NAME DISPLAY VERSION REPLACES PHASE
packageserver Package Server 0.19.1 Installing
Requirement Status:
Group: operators.coreos.com
Kind: ClusterServiceVersion
Message: CSV minKubeVersion (1.11.0) less than server version (v1.20.7-eks-d88609)
Name: packageserver
Status: Present
Version: v1alpha1
Group: apiregistration.k8s.io
Kind: APIService
Message:
Name: v1.packages.operators.coreos.com
Status: DeploymentFound
Version: v1
Dependents:
Group: rbac.authorization.k8s.io
Kind: PolicyRule
Message: cluster rule:{"verbs":["create","get"],"apiGroups":["authorization.k8s.io"],"resources":["subjectaccessreviews"]}
Status: Satisfied
Version: v1
Group: rbac.authorization.k8s.io
Kind: PolicyRule
Message: cluster rule:{"verbs":["get","list","watch"],"apiGroups":[""],"resources":["configmaps"]}
Status: Satisfied
Version: v1
Group: rbac.authorization.k8s.io
Kind: PolicyRule
Message: cluster rule:{"verbs":["get","list","watch"],"apiGroups":["operators.coreos.com"],"resources":["catalogsources"]}
Status: Satisfied
Version: v1
Group: rbac.authorization.k8s.io
Kind: PolicyRule
Message: cluster rule:{"verbs":["get","list"],"apiGroups":["packages.operators.coreos.com"],"resources":["packagemanifests"]}
Status: Satisfied
Version: v1
Group:
Kind: ServiceAccount
Message:
Name: olm-operator-serviceaccount
Status: Present
Version: v1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal RequirementsUnknown 52m operator-lifecycle-manager requirements not yet checked
Normal AllRequirementsMet 42m (x6 over 52m) operator-lifecycle-manager all requirements found, attempting install
Normal NeedsReinstall 42m (x4 over 47m) operator-lifecycle-manager APIServices not installed
Normal InstallSucceeded 42m (x6 over 52m) operator-lifecycle-manager waiting for install components to report healthy
Normal InstallWaiting 42m (x5 over 52m) operator-lifecycle-manager APIServices not installed
Warning InstallCheckFailed 2m17s (x19 over 47m) operator-lifecycle-manager install timeout
kl -n olm olm-operator-9bb87877b-zmr7f --tail=50 -f
time="2021-12-08T20:17:00Z" level=info msg="install strategy successful" csv=packageserver id=bVi6s namespace=olm phase=Installing strategy=deployment
time="2021-12-08T20:17:00Z" level=info msg="install strategy successful" csv=packageserver id=Su0tW namespace=olm phase=Installing strategy=deployment
time="2021-12-08T20:17:01Z" level=warning msg="install timed out" csv=packageserver id=AzvRi namespace=olm phase=Installing
I1208 20:17:01.095196 1 event.go:282] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"olm", Name:"packageserver", UID:"36910f88-c948-4d33-91e9-940ba5361b1f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"147367595", FieldPath:""}): type: 'Warning' reason: 'InstallCheckFailed' install timeout
time="2021-12-08T20:17:01Z" level=warning msg="install timed out" csv=packageserver id=eYZNN namespace=olm phase=Installing
I1208 20:17:01.250682 1 event.go:282] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"olm", Name:"packageserver", UID:"36910f88-c948-4d33-91e9-940ba5361b1f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"147367595", FieldPath:""}): type: 'Warning' reason: 'InstallCheckFailed' install timeout
time="2021-12-08T20:17:01Z" level=info msg="error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"packageserver\": the object has been modified; please apply your changes to the latest version and try again" csv=packageserver id=SldDi namespace=olm phase=Installing
E1208 20:17:01.267190 1 queueinformer_operator.go:290] sync {"update" "olm/packageserver"} failed: error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com "packageserver": the object has been modified; please apply your changes to the latest version and try again
time="2021-12-08T20:17:01Z" level=warning msg="needs reinstall: APIServices not installed" csv=packageserver id=kLQmP namespace=olm phase=Failed strategy=deployment
I1208 20:17:01.445074 1 event.go:282] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"olm", Name:"packageserver", UID:"36910f88-c948-4d33-91e9-940ba5361b1f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"147369389", FieldPath:""}): type: 'Normal' reason: 'NeedsReinstall' APIServices not installed
time="2021-12-08T20:17:01Z" level=warning msg="needs reinstall: APIServices not installed" csv=packageserver id=owVzw namespace=olm phase=Failed strategy=deployment
I1208 20:17:01.646871 1 event.go:282] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"olm", Name:"packageserver", UID:"36910f88-c948-4d33-91e9-940ba5361b1f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"147369389", FieldPath:""}): type: 'Normal' reason: 'NeedsReinstall' APIServices not installed
time="2021-12-08T20:17:01Z" level=info msg="error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"packageserver\": the object has been modified; please apply your changes to the latest version and try again" csv=packageserver id=ePxEX namespace=olm phase=Failed
E1208 20:17:01.655655 1 queueinformer_operator.go:290] sync {"update" "olm/packageserver"} failed: error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com "packageserver": the object has been modified; please apply your changes to the latest version and try again
time="2021-12-08T20:17:01Z" level=info msg="scheduling ClusterServiceVersion for install" csv=packageserver id=4rhbL namespace=olm phase=Pending
I1208 20:17:01.748610 1 event.go:282] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"olm", Name:"packageserver", UID:"36910f88-c948-4d33-91e9-940ba5361b1f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"147369396", FieldPath:""}): type: 'Normal' reason: 'AllRequirementsMet' all requirements found, attempting install
time="2021-12-08T20:17:01Z" level=info msg="scheduling ClusterServiceVersion for install" csv=packageserver id=h2fyb namespace=olm phase=Pending
I1208 20:17:01.844415 1 event.go:282] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"olm", Name:"packageserver", UID:"36910f88-c948-4d33-91e9-940ba5361b1f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"147369396", FieldPath:""}): type: 'Normal' reason: 'AllRequirementsMet' all requirements found, attempting install
time="2021-12-08T20:17:01Z" level=info msg="error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"packageserver\": the object has been modified; please apply your changes to the latest version and try again" csv=packageserver id=q0q+v namespace=olm phase=Pending
E1208 20:17:01.852175 1 queueinformer_operator.go:290] sync {"update" "olm/packageserver"} failed: error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com "packageserver": the object has been modified; please apply your changes to the latest version and try again
time="2021-12-08T20:17:01Z" level=warning msg="reusing existing cert packageserver-service-cert"
I1208 20:17:02.010812 1 event.go:282] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"olm", Name:"packageserver", UID:"36910f88-c948-4d33-91e9-940ba5361b1f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"147369400", FieldPath:""}): type: 'Normal' reason: 'InstallSucceeded' waiting for install components to report healthy
time="2021-12-08T20:17:02Z" level=warning msg="reusing existing cert packageserver-service-cert"
I1208 20:17:02.191973 1 event.go:282] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"olm", Name:"packageserver", UID:"36910f88-c948-4d33-91e9-940ba5361b1f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"147369400", FieldPath:""}): type: 'Normal' reason: 'InstallSucceeded' waiting for install components to report healthy
time="2021-12-08T20:17:02Z" level=info msg="error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"packageserver\": the object has been modified; please apply your changes to the latest version and try again" csv=packageserver id=FSdCV namespace=olm phase=InstallReady
E1208 20:17:02.202441 1 queueinformer_operator.go:290] sync {"update" "olm/packageserver"} failed: error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com "packageserver": the object has been modified; please apply your changes to the latest version and try again
time="2021-12-08T20:17:02Z" level=info msg="install strategy successful" csv=packageserver id=UY7It namespace=olm phase=Installing strategy=deployment
I1208 20:17:02.221591 1 event.go:282] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"olm", Name:"packageserver", UID:"36910f88-c948-4d33-91e9-940ba5361b1f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"147369412", FieldPath:""}): type: 'Normal' reason: 'InstallWaiting' APIServices not installed
time="2021-12-08T20:17:02Z" level=info msg="install strategy successful" csv=packageserver id=PqBQQ namespace=olm phase=Installing strategy=deployment
I1208 20:17:02.248508 1 event.go:282] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"olm", Name:"packageserver", UID:"36910f88-c948-4d33-91e9-940ba5361b1f", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"147369412", FieldPath:""}): type: 'Normal' reason: 'InstallWaiting' APIServices not installed
time="2021-12-08T20:17:02Z" level=info msg="error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"packageserver\": the object has been modified; please apply your changes to the latest version and try again" csv=packageserver id=kebtg namespace=olm phase=Installing
E1208 20:17:02.258585 1 queueinformer_operator.go:290] sync {"update" "olm/packageserver"} failed: error updating ClusterServiceVersion status: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com "packageserver": the object has been modified; please apply your changes to the latest version and try again
time="2021-12-08T20:17:02Z" level=info msg="install strategy successful" csv=packageserver id=x5s4G namespace=olm phase=Installing strategy=deployment
Environment
- operator-lifecycle-manager version: 0.19.1
- Kubernetes version information: 1.20 EKS
Istio 1.11.3
kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T21:16:14Z", GoVersion:"go1.16.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.7-eks-d88609", GitCommit:"d886092805d5cc3a47ed5cf0c43de38ce442dfcb", GitTreeState:"clean", BuildDate:"2021-07-31T00:29:12Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"}
- Kubernetes cluster kind:
Possible Solution
Additional context Add any other context about the problem here.
Possibly Related https://github.com/operator-framework/operator-lifecycle-manager/issues/1368 https://github.com/operator-framework/operator-lifecycle-manager/issues/2343 https://github.com/operator-framework/operator-lifecycle-manager/issues/2234
Hi there, Unfortunately, there is not enough information on the description to identify what the root cause is. I would like to ask for the pod/deployment yaml blob for the packageserver on the olm namespace.
Hey, had a similar issue using cilium, pushed a fix into my fork. Problem essentially is that EKS can't connect to the packageserver API, which I've solved by using hostNetwork on these pods.: https://github.com/operator-framework/operator-lifecycle-manager/compare/master...deinstapel:operator-lifecycle-manager:v0.23.1-fix
