operator-lifecycle-manager [FLAKE] updates multiple intermediates

Description

Subscription 
  updates multiple intermediates
  /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:511

[BeforeEach] Subscription
  /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:57
created the subscription-e2e-4smn9 testing namespace
[It] updates multiple intermediates
  /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:511
Creating catalog source mock-nginx-ff9rg in namespace subscription-e2e-4smn9...
Catalog source mock-nginx-ff9rg created
waiting for catalog pod mock-nginx-ff9rg to be available (for sync) - NO_CONNECTION
waiting for catalog pod mock-nginx-ff9rg to be available (for sync) - IDLE
waiting for catalog pod mock-nginx-ff9rg to be available (for sync) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (for sync) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (for sync) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (for sync) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (for sync) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (for sync) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (for sync) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (for sync) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (for sync) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (for sync) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (for sync) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (for sync) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (for sync) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (for sync) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (for sync) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (for sync) - TRANSIENT_FAILURE
catalog mock-nginx-ff9rg pod with address mock-nginx-ff9rg.subscription-e2e-4smn9.svc:50051
08:35:30.2578: UpgradePending (nginx-a): &ObjectReference{Kind:InstallPlan,Namespace:subscription-e2e-4smn9,Name:install-rdmww,UID:372ffcf1-e9c3-4590-8cea-ac6b623252f7,APIVersion:operators.coreos.com/v1alpha1,ResourceVersion:2003,FieldPath:,}
waiting for sub-nginx-dnl2n to have installplan ref
nginx-a - Installing (InstallWaiting): installing: waiting for deployment dep4qq24 to become ready: deployment "dep4qq24" not available: Deployment does not have minimum availability.
nginx-a - Succeeded (InstallSucceeded): install strategy completed with no errors
catalog mock-nginx-ff9rg pod with address mock-nginx-ff9rg.subscription-e2e-4smn9.svc:50051
waiting for catalog pod mock-nginx-ff9rg to be available (after catalog update) - READY
waiting for catalog pod mock-nginx-ff9rg to be available (after catalog update) - IDLE
waiting for catalog pod mock-nginx-ff9rg to be available (after catalog update) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (after catalog update) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (after catalog update) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (after catalog update) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (after catalog update) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (after catalog update) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (after catalog update) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (after catalog update) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (after catalog update) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (after catalog update) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (after catalog update) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (after catalog update) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (after catalog update) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (after catalog update) - TRANSIENT_FAILURE
waiting for catalog pod mock-nginx-ff9rg to be available (after catalog update) - TRANSIENT_FAILURE
catalog updated
nginx-b -  (): 
nginx-b - Installing (InstallWaiting): installing: waiting for deployment dep6kjmd to become ready: deployment "dep6kjmd" not available: Deployment does not have minimum availability.
nginx-b - Installing (InstallWaiting): installing: waiting for deployment dep6kjmd to become ready: deployment "dep6kjmd" not available: Deployment does not have minimum availability.
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
nginx-b - Succeeded (InstallSucceeded): install strategy completed with no errors
Deleting config map mock-nginx-ff9rg-configmap...
Deleting catalog source mock-nginx-ff9rg...
waiting for the catalog source mock-nginx-ff9rg-tgh55 pod to be deleted...
waiting for the catalog source mock-nginx-ff9rg-tgh55 pod to be deleted...
waiting for the catalog source mock-nginx-ff9rg-tgh55 pod to be deleted...
waiting for the catalog source mock-nginx-ff9rg-tgh55 pod to be deleted...
waiting for the catalog source mock-nginx-ff9rg-tgh55 pod to be deleted...
waiting for the catalog source mock-nginx-ff9rg-tgh55 pod to be deleted...
waiting for the catalog source mock-nginx-ff9rg-tgh55 pod to be deleted...
waiting for the catalog source mock-nginx-ff9rg-tgh55 pod to be deleted...
waiting for the catalog source mock-nginx-ff9rg-tgh55 pod to be deleted...
waiting for the catalog source mock-nginx-ff9rg-tgh55 pod to be deleted...
waiting for the catalog source mock-nginx-ff9rg-tgh55 pod to be deleted...
waiting for the catalog source mock-nginx-ff9rg-tgh55 pod to be deleted...
[AfterEach] Subscription
  /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:61
collecting the subscription-e2e-4smn9 namespace artifacts as the 'updates multiple intermediates' test case failed
collecting logs in the ./artifacts/ artifacts directory
Storing the test artifact output in the artifacts/subscription-e2e-4smn9 directory
Collecting get catalogsources -o yaml output...
Collecting get subscriptions -o yaml output...
Collecting get operatorgroups -o yaml output...
Collecting get clusterserviceversions -o yaml output...
Collecting get installplans -o yaml output...
Collecting get pods -o wide output...
Collecting get events --sort-by .lastTimestamp output...
tearing down the subscription-e2e-4smn9 namespace


• Failure [154.091 seconds]
Subscription
/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:52
  updates multiple intermediates [It]
  /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:511

  Timed out after 60.001s.
  Expected
      <bool>: false
  to be true

  /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/csv_e2e_test.go:4728

  Full Stack Trace
  github.com/operator-framework/operator-lifecycle-manager/test/e2e.awaitCSV({0x32d2518, 0xc000e6f170}, {0xc001a0f440, 0x16}, {0x2ee1698, 0x7}, 0xc0006ab820)
  	/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/csv_e2e_test.go:4728 +0x1db
  github.com/operator-framework/operator-lifecycle-manager/test/e2e.glob..func22.9.1({0xc000fa6000, 0xc000c7[976](https://github.com/operator-framework/operator-lifecycle-manager/runs/5213111320?check_suite_focus=true#step:4:976)8})
  	/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:563 +0xe5
  created by github.com/operator-framework/operator-lifecycle-manager/test/e2e.glob..func22.9
  	/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/subscription_e2e_test.go:559 +0x106f

Feb 16 '22 08:02 perdasilva

This test installs csvA first. Then it updates the catalog to make upgrade csvA -> csvB -> csvC happen. When it works, the csvA is replaced by csvB successfully and then csvA is deleted during the syncResolvingNamespace is still repeatedly called because of the series of subscription updates. When it failed, the deletion of the csvA didn't happen soon and no more syncResolvingNamespace call happened. The o.resolver.ResolveSteps(namespace) call in syncResolvingNamespace fails with

"resolution failed" error="constraints not satisfiable: @existing/subscription-e2e-4smn9//nginx-b, @existing/subscription-e2e-4smn9//nginx-a and mock-nginx-ff9rg/subscription-e2e-4smn9/stable/nginx-c originate from package nginx-n99ss, clusterserviceversion nginx-a exists and is not referenced by a subscription, subscription sub-nginx-dnl2n exists, subscription sub-nginx-dnl2n requires at least one of mock-nginx-ff9rg/subscription-e2e-4smn9/stable/nginx-c or @existing/subscription-e2e-4smn9//nginx-b" id=Bdo7I namespace=subscription-e2e-4smn9

until the csvA is deleted (or marked deleting). So the replacing from csvB to csvC doesn't happen.

How should this be fixed?

Feb 23 '22 21:02 akihikokuroda

Could the e2e be right and the problem be with the operator?

May 10 '22 07:05 perdasilva

operator-lifecycle-manager operator-lifecycle-manager copied to clipboard

[FLAKE] updates multiple intermediates

Description

operator-lifecycle-manager
operator-lifecycle-manager copied to clipboard