karmada
karmada copied to clipboard
karmada-io
deployment replicas syncer testing
Which jobs are flaking:
Which test(s) are flaking:
https://github.com/karmada-io/karmada/actions/runs/9788476477/job/27026725265
Reason for failure:
Anything else we need to know:
Timeline >>
STEP: Creating PropagationPolicy(karmadatest-m5dpb/deploy-g69xd) @ 07/04/24 04:42:38.3
STEP: Creating Deployment(karmadatest-m5dpb/deploy-g69xd) @ 07/04/24 04:42:38.31
STEP: Creating HPA(karmadatest-m5dpb/deploy-g69xd) @ 07/04/24 04:42:38.315
STEP: step1: propagate 4 replicas to two clusters @ 07/04/24 04:42:38.323
STEP: Waiting for deployment(karmadatest-m5dpb/deploy-g69xd) synced on member clusters @ 07/04/24 04:42:38.323
STEP: Waiting for deployment(karmadatest-m5dpb/deploy-g69xd) synced on member clusters @ 07/04/24 04:42:43.332
STEP: step2: hpa scale each member cluster replicas from 2 to 3 @ 07/04/24 04:42:43.349
STEP: Updating HPA(karmadatest-m5dpb/deploy-g69xd) @ 07/04/24 04:42:43.349
STEP: Waiting for deployment(karmadatest-m5dpb/deploy-g69xd) synced on member clusters @ 07/04/24 04:42:43.369
STEP: Waiting for deployment(karmadatest-m5dpb/deploy-g69xd) synced on member clusters @ 07/04/24 04:42:58.399
STEP: step3: manually add deployment template replicas from 6 to 10 @ 07/04/24 04:42:58.422
STEP: Updating Deployment(karmadatest-m5dpb/deploy-g69xd)'s replicas to 10 @ 07/04/24 04:42:58.422
STEP: Waiting for deployment(karmadatest-m5dpb/deploy-g69xd) synced on member clusters @ 07/04/24 04:42:58.447
STEP: Waiting for deployment(karmadatest-m5dpb/deploy-g69xd) synced on member clusters @ 07/04/24 04:42:58.461
[FAILED] in [It] - /home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:247 @ 07/04/24 04:43:58.472
STEP: Removing PropagationPolicy(karmadatest-m5dpb/deploy-g69xd) @ 07/04/24 04:43:58.473
STEP: Removing Deployment(karmadatest-m5dpb/deploy-g69xd) @ 07/04/24 04:43:58.479
STEP: Removing HPA(karmadatest-m5dpb/deploy-g69xd) @ 07/04/24 04:43:58.483
STEP: Check if deployment(karmadatest-m5dpb/deploy-g69xd) disappears on member clusters @ 07/04/24 04:43:58.489
<< Timeline
[FAILED] Timed out after 60.000s.
Expected
<bool>: false
to equal
<bool>: true
In [It] at: /home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:247 @ 07/04/24 04:43:58.472
Full Stack Trace
github.com/karmada-io/karmada/test/e2e.assertDeploymentTemplateReplicas({0xc00051eca8, 0x11}, {0xc0001f5744, 0xc}, 0x6)
/home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:247 +0x257
github.com/karmada-io/karmada/test/e2e.init.func13.4.2.3()
/home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:142 +0x1de
github.com/karmada-io/karmada/test/e2e.init.func13.4.2()
/home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:139 +0x2fa
the log of controller-manager is:
controller-manager.log
$ cat controller-manager.log | grep deployment_replicas_syncer_controller.go | grep -A 200 '2024-07-04T04:42:38.348681435Z stderr F I0704 04:42:38.348611'
2024-07-04T04:42:38.348681435Z stderr F I0704 04:42:38.348611 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:42:38.349314846Z stderr F I0704 04:42:38.349251 1 deployment_replicas_syncer_controller.go:163] wait until scheduler observed generation (0) equal to generation in binding (1)
2024-07-04T04:42:38.633299953Z stderr F I0704 04:42:38.633098 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:42:38.633305023Z stderr F I0704 04:42:38.633147 1 deployment_replicas_syncer_controller.go:190] wait until aggregated status replicas of cluster member1 collected
2024-07-04T04:42:38.75664433Z stderr F I0704 04:42:38.756584 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:42:38.756797706Z stderr F I0704 04:42:38.756727 1 deployment_replicas_syncer_controller.go:129] replicas in spec field (4) already equal to in status field (4)
2024-07-04T04:42:39.350750574Z stderr F I0704 04:42:39.350615 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:42:39.350765843Z stderr F I0704 04:42:39.350711 1 deployment_replicas_syncer_controller.go:129] replicas in spec field (4) already equal to in status field (4)
2024-07-04T04:42:41.788647528Z stderr F I0704 04:42:41.788411 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-75h5m/deploy-j9vtk
2024-07-04T04:42:41.788685509Z stderr F I0704 04:42:41.788652 1 deployment_replicas_syncer_controller.go:163] wait until scheduler observed generation (0) equal to generation in binding (1)
2024-07-04T04:42:42.032515467Z stderr F I0704 04:42:42.032456 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-75h5m/deploy-j9vtk
2024-07-04T04:42:42.034628338Z stderr F I0704 04:42:42.034574 1 deployment_replicas_syncer_controller.go:190] wait until aggregated status replicas of cluster member2 collected
2024-07-04T04:42:42.058527425Z stderr F I0704 04:42:42.058452 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-75h5m/deploy-j9vtk
2024-07-04T04:42:42.058737587Z stderr F I0704 04:42:42.058661 1 deployment_replicas_syncer_controller.go:129] replicas in spec field (2) already equal to in status field (2)
2024-07-04T04:42:42.793182239Z stderr F I0704 04:42:42.792535 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-75h5m/deploy-j9vtk
2024-07-04T04:42:42.79323648Z stderr F I0704 04:42:42.792589 1 deployment_replicas_syncer_controller.go:129] replicas in spec field (2) already equal to in status field (2)
2024-07-04T04:42:46.790275125Z stderr F I0704 04:42:46.789925 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-75h5m/deploy-j9vtk
2024-07-04T04:42:46.790279754Z stderr F I0704 04:42:46.790064 1 deployment_replicas_syncer_controller.go:156] wait until replicas of binding (2) equal to replicas of deployment (1)
2024-07-04T04:42:46.941194503Z stderr F I0704 04:42:46.941089 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-75h5m/deploy-j9vtk
2024-07-04T04:42:46.941763033Z stderr F I0704 04:42:46.941702 1 deployment_replicas_syncer_controller.go:129] replicas in spec field (1) already equal to in status field (1)
2024-07-04T04:42:47.790522419Z stderr F I0704 04:42:47.790373 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-75h5m/deploy-j9vtk
2024-07-04T04:42:47.790545312Z stderr F I0704 04:42:47.790446 1 deployment_replicas_syncer_controller.go:129] replicas in spec field (1) already equal to in status field (1)
2024-07-04T04:42:54.537885193Z stderr F I0704 04:42:54.536588 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:42:54.601312782Z stderr F I0704 04:42:54.601238 1 deployment_replicas_syncer_controller.go:146] successfully update deployment (karmadatest-m5dpb/deploy-g69xd) replicas from 4 to 5
2024-07-04T04:42:54.601813556Z stderr F I0704 04:42:54.601572 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:42:54.60183161Z stderr F I0704 04:42:54.601651 1 deployment_replicas_syncer_controller.go:129] replicas in spec field (5) already equal to in status field (5)
2024-07-04T04:42:54.631012895Z stderr F I0704 04:42:54.630939 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:42:54.631175348Z stderr F I0704 04:42:54.631121 1 deployment_replicas_syncer_controller.go:156] wait until replicas of binding (4) equal to replicas of deployment (5)
2024-07-04T04:42:55.633735224Z stderr F I0704 04:42:55.633616 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:42:55.658987001Z stderr F I0704 04:42:55.658890 1 deployment_replicas_syncer_controller.go:146] successfully update deployment (karmadatest-m5dpb/deploy-g69xd) replicas from 5 to 6
2024-07-04T04:42:55.65915775Z stderr F I0704 04:42:55.659098 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:42:55.659294865Z stderr F I0704 04:42:55.659247 1 deployment_replicas_syncer_controller.go:129] replicas in spec field (6) already equal to in status field (6)
2024-07-04T04:42:58.44880422Z stderr F I0704 04:42:58.448717 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:42:58.45040973Z stderr F I0704 04:42:58.450326 1 deployment_replicas_syncer_controller.go:156] wait until replicas of binding (6) equal to replicas of deployment (10)
2024-07-04T04:42:59.483885582Z stderr F I0704 04:42:59.475252 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:42:59.483954461Z stderr F I0704 04:42:59.483873 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:42:59.483962024Z stderr F I0704 04:42:59.483914 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:42:59.48402365Z stderr F I0704 04:42:59.483939 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:00.511409144Z stderr F I0704 04:43:00.511009 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:00.511540069Z stderr F I0704 04:43:00.511491 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:01.536768598Z stderr F I0704 04:43:01.534993 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:01.536789396Z stderr F I0704 04:43:01.535103 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:02.579765844Z stderr F I0704 04:43:02.579690 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:02.600448542Z stderr F I0704 04:43:02.600279 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:03.643271921Z stderr F I0704 04:43:03.602373 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:03.643306346Z stderr F I0704 04:43:03.602467 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:04.615004062Z stderr F I0704 04:43:04.609571 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:04.615016696Z stderr F I0704 04:43:04.609625 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:05.610707101Z stderr F I0704 04:43:05.610588 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:05.610844317Z stderr F I0704 04:43:05.610786 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:06.612604682Z stderr F I0704 04:43:06.612479 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:06.612708105Z stderr F I0704 04:43:06.612591 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:07.617791715Z stderr F I0704 04:43:07.617742 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:07.618290706Z stderr F I0704 04:43:07.618243 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:08.995610749Z stderr F I0704 04:43:08.623748 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:08.995760749Z stderr F I0704 04:43:08.992365 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:09.993049138Z stderr F I0704 04:43:09.992990 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:09.993207694Z stderr F I0704 04:43:09.993163 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:10.994547653Z stderr F I0704 04:43:10.994465 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:10.994724032Z stderr F I0704 04:43:10.994683 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:11.994980148Z stderr F I0704 04:43:11.994878 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:11.995400993Z stderr F I0704 04:43:11.995050 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:13.001582738Z stderr F I0704 04:43:13.001435 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:13.009395901Z stderr F I0704 04:43:13.009305 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:14.009906666Z stderr F I0704 04:43:14.009715 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:14.032956097Z stderr F I0704 04:43:14.009832 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:15.010596957Z stderr F I0704 04:43:15.010545 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:15.012511178Z stderr F I0704 04:43:15.010688 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:16.013161271Z stderr F I0704 04:43:16.013056 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:16.013324646Z stderr F I0704 04:43:16.013279 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:17.013842394Z stderr F I0704 04:43:17.013733 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:17.014036596Z stderr F I0704 04:43:17.013971 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:18.206775381Z stderr F I0704 04:43:18.199304 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:18.20678021Z stderr F I0704 04:43:18.199394 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:19.225669668Z stderr F I0704 04:43:19.225582 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:19.230145606Z stderr F I0704 04:43:19.228308 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:20.230956422Z stderr F I0704 04:43:20.230853 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:20.231144663Z stderr F I0704 04:43:20.231090 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:21.232135366Z stderr F I0704 04:43:21.232021 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:21.232332634Z stderr F I0704 04:43:21.232274 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:22.233620913Z stderr F I0704 04:43:22.233483 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:22.233655998Z stderr F I0704 04:43:22.233584 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:23.234693049Z stderr F I0704 04:43:23.234511 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:23.235077005Z stderr F I0704 04:43:23.235007 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:24.238761179Z stderr F I0704 04:43:24.238678 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:24.238910217Z stderr F I0704 04:43:24.238831 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:25.239605969Z stderr F I0704 04:43:25.239499 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:25.23981552Z stderr F I0704 04:43:25.239759 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:26.240888078Z stderr F I0704 04:43:26.240759 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:26.241134488Z stderr F I0704 04:43:26.241048 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:27.24218126Z stderr F I0704 04:43:27.242049 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:27.242206257Z stderr F I0704 04:43:27.242146 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:28.253078689Z stderr F I0704 04:43:28.245814 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:28.253088727Z stderr F I0704 04:43:28.251382 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:29.256729894Z stderr F I0704 04:43:29.256600 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:29.256766452Z stderr F I0704 04:43:29.256703 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:30.283301444Z stderr F I0704 04:43:30.283119 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:30.283443859Z stderr F I0704 04:43:30.283382 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:31.283734007Z stderr F I0704 04:43:31.283598 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:31.284009511Z stderr F I0704 04:43:31.283944 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:32.284739839Z stderr F I0704 04:43:32.284626 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:32.284822934Z stderr F I0704 04:43:32.284769 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:33.28553309Z stderr F I0704 04:43:33.285426 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:33.285755475Z stderr F I0704 04:43:33.285703 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:34.286656118Z stderr F I0704 04:43:34.286518 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:34.286816767Z stderr F I0704 04:43:34.286764 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:35.294992759Z stderr F I0704 04:43:35.294006 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:35.295366987Z stderr F I0704 04:43:35.294119 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:36.29508643Z stderr F I0704 04:43:36.294961 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:36.29512414Z stderr F I0704 04:43:36.295045 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:37.295342322Z stderr F I0704 04:43:37.295189 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:37.295365095Z stderr F I0704 04:43:37.295264 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:38.295741609Z stderr F I0704 04:43:38.295643 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:38.295814394Z stderr F I0704 04:43:38.295753 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:39.296305295Z stderr F I0704 04:43:39.296180 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:39.296399281Z stderr F I0704 04:43:39.296276 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:40.29744568Z stderr F I0704 04:43:40.297331 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:40.297479062Z stderr F I0704 04:43:40.297401 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:41.298582571Z stderr F I0704 04:43:41.298433 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:41.299078476Z stderr F I0704 04:43:41.298987 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:42.299877223Z stderr F I0704 04:43:42.299756 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:42.29991297Z stderr F I0704 04:43:42.299844 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:43.300724312Z stderr F I0704 04:43:43.300619 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:43.300750721Z stderr F I0704 04:43:43.300688 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:44.301558372Z stderr F I0704 04:43:44.301432 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:44.3015843Z stderr F I0704 04:43:44.301503 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:45.30220186Z stderr F I0704 04:43:45.302082 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:45.302316327Z stderr F I0704 04:43:45.302211 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:46.303055614Z stderr F I0704 04:43:46.302959 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:46.303111756Z stderr F I0704 04:43:46.303032 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:47.303599296Z stderr F I0704 04:43:47.303435 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:47.303653994Z stderr F I0704 04:43:47.303544 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:48.304638729Z stderr F I0704 04:43:48.304493 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:48.304683642Z stderr F I0704 04:43:48.304616 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:49.309119683Z stderr F I0704 04:43:49.304715 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:49.309310148Z stderr F I0704 04:43:49.309258 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:50.310571176Z stderr F I0704 04:43:50.310451 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:50.310641647Z stderr F I0704 04:43:50.310574 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:51.311769002Z stderr F I0704 04:43:51.311664 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:51.311797284Z stderr F I0704 04:43:51.311739 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:52.312894451Z stderr F I0704 04:43:52.312782 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:52.312931872Z stderr F I0704 04:43:52.312877 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:53.313885509Z stderr F I0704 04:43:53.313756 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:53.313998073Z stderr F I0704 04:43:53.313907 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:54.324535097Z stderr F I0704 04:43:54.324396 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:54.32454234Z stderr F I0704 04:43:54.324495 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:55.32563716Z stderr F I0704 04:43:55.325532 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:55.325670692Z stderr F I0704 04:43:55.325606 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:56.326297593Z stderr F I0704 04:43:56.326190 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:56.326351323Z stderr F I0704 04:43:56.326284 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:57.327330613Z stderr F I0704 04:43:57.327239 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:57.327368715Z stderr F I0704 04:43:57.327317 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:58.328475923Z stderr F I0704 04:43:58.328375 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:58.328504757Z stderr F I0704 04:43:58.328457 1 deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
2024-07-04T04:43:59.328768022Z stderr F I0704 04:43:59.328640 1 deployment_replicas_syncer_controller.go:98] Reconciling for Deployment karmadatest-m5dpb/deploy-g69xd
2024-07-04T04:43:59.328786577Z stderr F I0704 04:43:59.328685 1 deployment_replicas_syncer_controller.go:107] no need to update deployment replicas for binding not found
the key error log is deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected, the corresponding code is:
https://github.com/karmada-io/karmada/blob/c4a81bf36cfc66b357e01d7ea8863d0c4725e139/pkg/controllers/deploymentreplicassyncer/deployment_replicas_syncer_controller.go#L175-L181
So, when we manually modify deployment template's spec.replicas from 6 to 10, the controller is expected to update it back to 6 on condition of member clusters' status collected, however, it's strange that binding.Status.AggregatedStatus["member1"].Status is nil, which caused wait loop keep executing.
how many times does this error occur?
Just found it once on the master branch.
So, when we manually modify deployment template's
spec.replicasfrom 6 to 10, the controller is expected to update it back to 6 on condition of member clusters' status collected, however, it's strange thatbinding.Status.AggregatedStatus["member1"].Statusis nil, which caused wait loop keep executing.
Considering the low probability of occurrence and there are no major errors in logic, shall we do thing to handle it? @RainbowMango
Found another failure at PR 5146 with the similar output: https://github.com/karmada-io/karmada/actions/runs/9917886710/job/27401983740?pr=5146
[FAILED] in [It] - /home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:247 @ 07/13/24 06:38:03.994
STEP: Removing PropagationPolicy(karmadatest-8jmgv/deploy-6pbdk) @ 07/13/24 06:38:03.995
STEP: Removing Deployment(karmadatest-8jmgv/deploy-6pbdk) @ 07/13/24 06:38:03.999
STEP: Removing HPA(karmadatest-8jmgv/deploy-6pbdk) @ 07/13/24 06:38:04.002
STEP: Check if deployment(karmadatest-8jmgv/deploy-6pbdk) disappears on member clusters @ 07/13/24 06:38:04.009
<< Timeline
[FAILED] Timed out after 60.001s.
Expected
<bool>: false
to equal
<bool>: true
In [It] at: /home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:247 @ 07/13/24 06:38:03.994
Full Stack Trace
github.com/karmada-io/karmada/test/e2e.assertDeploymentTemplateReplicas({0xc000b22798, 0x11}, {0xc000899224, 0xc}, 0x6)
/home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:247 +0x257
github.com/karmada-io/karmada/test/e2e.init.func13.4.2.3()
/home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:142 +0x1de
github.com/karmada-io/karmada/test/e2e.init.func13.4.2()
/home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:139 +0x2fa
Another failing test found on master: https://github.com/karmada-io/karmada/actions/runs/9966619439/job/27539114570#step:6:3178
[FAILED] in [It] - /home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:247 @ 07/17/24 01:49:53.305
STEP: Removing PropagationPolicy(karmadatest-969hj/deploy-b2gm9) @ 07/17/24 01:49:53.305
STEP: Removing Deployment(karmadatest-969hj/deploy-b2gm9) @ 07/17/24 01:49:53.311
STEP: Removing HPA(karmadatest-969hj/deploy-b2gm9) @ 07/17/24 01:49:53.319
STEP: Check if deployment(karmadatest-969hj/deploy-b2gm9) disappears on member clusters @ 07/17/24 01:49:53.324
<< Timeline
[FAILED] Timed out after 60.000s.
Expected
<bool>: false
to equal
<bool>: true
In [It] at: /home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:247 @ 07/17/24 01:49:53.305
Full Stack Trace
github.com/karmada-io/karmada/test/e2e.assertDeploymentTemplateReplicas({0xc0003a7860, 0x11}, {0xc000e55b64, 0xc}, 0x6)
/home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:247 +0x257
github.com/karmada-io/karmada/test/e2e.init.func13.4.2.3()
/home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:142 +0x1de
github.com/karmada-io/karmada/test/e2e.init.func13.4.2()
/home/runner/work/karmada/karmada/test/e2e/deploymentreplicassyncer_test.go:139 +0x2fa
A similar case error: https://github.com/karmada-io/karmada/actions/runs/10105441369/job/27951974486?pr=5255
the problem lies in following abnormal status:
https://github.com/karmada-io/karmada/blob/835a446a74c8b3bf1b8ccca710c83512f65019d4/pkg/controllers/deploymentreplicassyncer/deployment_replicas_syncer_controller.go#L177-L181
but I don't know why binding.Status.AggregatedStatus[n].Status can be nil
to further explore this problem, I added some log in e2e test: #5263
Similar case with different error: https://github.com/karmada-io/karmada/actions/runs/10121815577/job/27993193537?pr=5263
Attachment: karmada_e2e_log_v1.29.0 (3).zip karmada_kind_log_v1.29.0 (1).zip
binding and deployment
{
"kind": "ResourceBinding",
"apiVersion": "work.karmada.io/v1alpha2",
"metadata": {
"name": "deploy-zp7hq-deployment",
"namespace": "karmadatest-5zbq7",
"uid": "0a8edbb4-365b-4d5c-aca2-d04b58fbb78a",
"resourceVersion": "9489",
"generation": 9,
"creationTimestamp": "2024-07-27T09:51:39Z",
"labels": {
"propagationpolicy.karmada.io/permanent-id": "c0e99a2d-eab7-4e55-851b-8589e467103c",
"resourcebinding.karmada.io/permanent-id": "32c43d16-140d-48aa-9422-9aa3bd50b6a7"
},
"annotations": {
"policy.karmada.io/applied-placement": "{\"clusterAffinity\":{\"clusterNames\":[\"member1\",\"member2\"]},\"clusterTolerations\":[{\"key\":\"cluster.karmada.io/not-ready\",\"operator\":\"Exists\",\"effect\":\"NoExecute\",\"tolerationSeconds\":30},{\"key\":\"cluster.karmada.io/unreachable\",\"operator\":\"Exists\",\"effect\":\"NoExecute\",\"tolerationSeconds\":30}],\"replicaScheduling\":{\"replicaSchedulingType\":\"Divided\",\"replicaDivisionPreference\":\"Weighted\",\"weightPreference\":{\"staticWeightList\":[{\"targetCluster\":{\"clusterNames\":[\"member1\"]},\"weight\":1},{\"targetCluster\":{\"clusterNames\":[\"member2\"]},\"weight\":1}]}}}",
"propagationpolicy.karmada.io/name": "deploy-zp7hq",
"propagationpolicy.karmada.io/namespace": "karmadatest-5zbq7"
},
"ownerReferences": [
{
"apiVersion": "apps/v1",
"kind": "Deployment",
"name": "deploy-zp7hq",
"uid": "705124a5-c541-427b-a9c6-041aa059b6a7",
"controller": true,
"blockOwnerDeletion": true
}
],
"finalizers": [
"karmada.io/binding-controller"
]
},
"spec": {
"resource": {
"apiVersion": "apps/v1",
"kind": "Deployment",
"namespace": "karmadatest-5zbq7",
"name": "deploy-zp7hq",
"uid": "705124a5-c541-427b-a9c6-041aa059b6a7",
"resourceVersion": "9442"
},
"replicaRequirements": {
"resourceRequest": {
"cpu": "10m"
}
},
"replicas": 10,
"clusters": [
{
"name": "member2",
"replicas": 5
},
{
"name": "member1",
"replicas": 5
}
],
"placement": {
"clusterAffinity": {
"clusterNames": [
"member1",
"member2"
]
},
"clusterTolerations": [
{
"key": "cluster.karmada.io/not-ready",
"operator": "Exists",
"effect": "NoExecute",
"tolerationSeconds": 30
},
{
"key": "cluster.karmada.io/unreachable",
"operator": "Exists",
"effect": "NoExecute",
"tolerationSeconds": 30
}
],
"replicaScheduling": {
"replicaSchedulingType": "Divided",
"replicaDivisionPreference": "Weighted",
"weightPreference": {
"staticWeightList": [
{
"targetCluster": {
"clusterNames": [
"member1"
]
},
"weight": 1
},
{
"targetCluster": {
"clusterNames": [
"member2"
]
},
"weight": 1
}
]
}
}
},
"schedulerName": "default-scheduler",
"conflictResolution": "Abort"
},
"status": {
"schedulerObservedGeneration": 9,
"lastScheduledTime": "2024-07-27T09:51:59Z",
"conditions": [
{
"type": "Scheduled",
"status": "True",
"lastTransitionTime": "2024-07-27T09:51:39Z",
"reason": "Success",
"message": "Binding has been scheduled successfully."
},
{
"type": "FullyApplied",
"status": "True",
"lastTransitionTime": "2024-07-27T09:51:59Z",
"reason": "FullyAppliedSuccess",
"message": "All works have been successfully applied"
}
],
"aggregatedStatus": [
{
"clusterName": "member1",
"status": {
"availableReplicas": 3,
"generation": 5,
"observedGeneration": 5,
"readyReplicas": 3,
"replicas": 3,
"resourceTemplateGeneration": 5,
"updatedReplicas": 3
},
"applied": true,
"health": "Healthy"
},
{
"clusterName": "member2",
"status": {
"availableReplicas": 3,
"generation": 5,
"observedGeneration": 5,
"readyReplicas": 3,
"replicas": 3,
"resourceTemplateGeneration": 5,
"updatedReplicas": 3
},
"applied": true,
"health": "Healthy"
}
]
}
}
{
"apiVersion": "apps/v1",
"kind": "Deployment",
"metadata": {
"annotations": {
"propagationpolicy.karmada.io/name": "deploy-zp7hq",
"propagationpolicy.karmada.io/namespace": "karmadatest-5zbq7"
},
"creationTimestamp": "2024-07-27T09:51:39Z",
"generation": 5,
"labels": {
"propagationpolicy.karmada.io/permanent-id": "c0e99a2d-eab7-4e55-851b-8589e467103c",
"resourcetemplate.karmada.io/retain-replicas": "true"
},
"name": "deploy-zp7hq",
"namespace": "karmadatest-5zbq7",
"resourceVersion": "9486",
"uid": "705124a5-c541-427b-a9c6-041aa059b6a7"
},
"spec": {
"progressDeadlineSeconds": 600,
"replicas": 10,
"revisionHistoryLimit": 10,
"selector": {
"matchLabels": {
"app": "nginx"
}
},
"strategy": {
"rollingUpdate": {
"maxSurge": "25%",
"maxUnavailable": "25%"
},
"type": "RollingUpdate"
},
"template": {
"metadata": {
"creationTimestamp": null,
"labels": {
"app": "nginx"
}
},
"spec": {
"containers": [
{
"image": "nginx:1.19.0",
"imagePullPolicy": "IfNotPresent",
"name": "nginx",
"ports": [
{
"containerPort": 80,
"protocol": "TCP"
}
],
"resources": {
"limits": {
"cpu": "100m"
},
"requests": {
"cpu": "10m"
}
},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File"
}
],
"dnsPolicy": "ClusterFirst",
"restartPolicy": "Always",
"schedulerName": "default-scheduler",
"securityContext": {},
"terminationGracePeriodSeconds": 30
}
}
},
"status": {
"availableReplicas": 3,
"observedGeneration": 5,
"readyReplicas": 3,
"replicas": 3,
"updatedReplicas": 3
}
}
{
"kind": "Work",
"apiVersion": "work.karmada.io/v1alpha1",
"metadata": {
"name": "deploy-zp7hq-784b697bfc",
"namespace": "karmada-es-member1",
"uid": "21620ad6-c017-4d6e-aa5f-452115b04e7c",
"resourceVersion": "9463",
"generation": 6,
"creationTimestamp": "2024-07-27T09:51:39Z",
"labels": {
"resourcebinding.karmada.io/permanent-id": "32c43d16-140d-48aa-9422-9aa3bd50b6a7",
"work.karmada.io/permanent-id": "9e2dee79-3455-4248-bdc7-59dc5f5cca4e"
},
"annotations": {
"resourcebinding.karmada.io/name": "deploy-zp7hq-deployment",
"resourcebinding.karmada.io/namespace": "karmadatest-5zbq7",
"work.karmada.io/conflict-resolution": "abort"
},
"finalizers": [
"karmada.io/execution-controller"
]
},
"spec": {
"workload": {
"manifests": [
{
"apiVersion": "apps/v1",
"kind": "Deployment",
"metadata": {
"annotations": {
"propagationpolicy.karmada.io/name": "deploy-zp7hq",
"propagationpolicy.karmada.io/namespace": "karmadatest-5zbq7",
"resourcebinding.karmada.io/name": "deploy-zp7hq-deployment",
"resourcebinding.karmada.io/namespace": "karmadatest-5zbq7",
"resourcetemplate.karmada.io/generation": "5",
"resourcetemplate.karmada.io/managed-annotations": "propagationpolicy.karmada.io/name,propagationpolicy.karmada.io/namespace,resourcebinding.karmada.io/name,resourcebinding.karmada.io/namespace,resourcetemplate.karmada.io/generation,resourcetemplate.karmada.io/managed-annotations,resourcetemplate.karmada.io/managed-labels,resourcetemplate.karmada.io/uid,work.karmada.io/conflict-resolution,work.karmada.io/name,work.karmada.io/namespace",
"resourcetemplate.karmada.io/managed-labels": "karmada.io/managed,propagationpolicy.karmada.io/permanent-id,resourcebinding.karmada.io/permanent-id,resourcetemplate.karmada.io/retain-replicas,work.karmada.io/permanent-id",
"resourcetemplate.karmada.io/uid": "705124a5-c541-427b-a9c6-041aa059b6a7",
"work.karmada.io/conflict-resolution": "abort",
"work.karmada.io/name": "deploy-zp7hq-784b697bfc",
"work.karmada.io/namespace": "karmada-es-member1"
},
"labels": {
"karmada.io/managed": "true",
"propagationpolicy.karmada.io/permanent-id": "c0e99a2d-eab7-4e55-851b-8589e467103c",
"resourcebinding.karmada.io/permanent-id": "32c43d16-140d-48aa-9422-9aa3bd50b6a7",
"resourcetemplate.karmada.io/retain-replicas": "true",
"work.karmada.io/permanent-id": "9e2dee79-3455-4248-bdc7-59dc5f5cca4e"
},
"name": "deploy-zp7hq",
"namespace": "karmadatest-5zbq7"
},
"spec": {
"progressDeadlineSeconds": 600,
"replicas": 5,
"revisionHistoryLimit": 10,
"selector": {
"matchLabels": {
"app": "nginx"
}
},
"strategy": {
"rollingUpdate": {
"maxSurge": "25%",
"maxUnavailable": "25%"
},
"type": "RollingUpdate"
},
"template": {
"metadata": {
"creationTimestamp": null,
"labels": {
"app": "nginx"
}
},
"spec": {
"containers": [
{
"image": "nginx:1.19.0",
"imagePullPolicy": "IfNotPresent",
"name": "nginx",
"ports": [
{
"containerPort": 80,
"protocol": "TCP"
}
],
"resources": {
"limits": {
"cpu": "100m"
},
"requests": {
"cpu": "10m"
}
},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File"
}
],
"dnsPolicy": "ClusterFirst",
"restartPolicy": "Always",
"schedulerName": "default-scheduler",
"securityContext": {},
"terminationGracePeriodSeconds": 30
}
}
}
}
]
}
},
"status": {
"conditions": [
{
"type": "Applied",
"status": "True",
"lastTransitionTime": "2024-07-27T09:51:39Z",
"reason": "AppliedSuccessful",
"message": "Manifest has been successfully applied"
}
],
"manifestStatuses": [
{
"identifier": {
"ordinal": 0,
"group": "apps",
"version": "v1",
"kind": "Deployment",
"resource": "",
"namespace": "karmadatest-5zbq7",
"name": "deploy-zp7hq"
},
"status": {
"availableReplicas": 3,
"generation": 5,
"observedGeneration": 5,
"readyReplicas": 3,
"replicas": 3,
"resourceTemplateGeneration": 5,
"updatedReplicas": 3
},
"health": "Healthy"
}
]
}
}
{
"kind": "Work",
"apiVersion": "work.karmada.io/v1alpha1",
"metadata": {
"name": "deploy-zp7hq-784b697bfc",
"namespace": "karmada-es-member2",
"uid": "cf964cfd-a051-4683-b6f8-caed365c0b46",
"resourceVersion": "9484",
"generation": 6,
"creationTimestamp": "2024-07-27T09:51:39Z",
"labels": {
"resourcebinding.karmada.io/permanent-id": "32c43d16-140d-48aa-9422-9aa3bd50b6a7",
"work.karmada.io/permanent-id": "8e251b6c-7e8a-4151-8808-7ba265b12fdf"
},
"annotations": {
"resourcebinding.karmada.io/name": "deploy-zp7hq-deployment",
"resourcebinding.karmada.io/namespace": "karmadatest-5zbq7",
"work.karmada.io/conflict-resolution": "abort"
},
"finalizers": [
"karmada.io/execution-controller"
]
},
"spec": {
"workload": {
"manifests": [
{
"apiVersion": "apps/v1",
"kind": "Deployment",
"metadata": {
"annotations": {
"propagationpolicy.karmada.io/name": "deploy-zp7hq",
"propagationpolicy.karmada.io/namespace": "karmadatest-5zbq7",
"resourcebinding.karmada.io/name": "deploy-zp7hq-deployment",
"resourcebinding.karmada.io/namespace": "karmadatest-5zbq7",
"resourcetemplate.karmada.io/generation": "5",
"resourcetemplate.karmada.io/managed-annotations": "propagationpolicy.karmada.io/name,propagationpolicy.karmada.io/namespace,resourcebinding.karmada.io/name,resourcebinding.karmada.io/namespace,resourcetemplate.karmada.io/generation,resourcetemplate.karmada.io/managed-annotations,resourcetemplate.karmada.io/managed-labels,resourcetemplate.karmada.io/uid,work.karmada.io/conflict-resolution,work.karmada.io/name,work.karmada.io/namespace",
"resourcetemplate.karmada.io/managed-labels": "karmada.io/managed,propagationpolicy.karmada.io/permanent-id,resourcebinding.karmada.io/permanent-id,resourcetemplate.karmada.io/retain-replicas,work.karmada.io/permanent-id",
"resourcetemplate.karmada.io/uid": "705124a5-c541-427b-a9c6-041aa059b6a7",
"work.karmada.io/conflict-resolution": "abort",
"work.karmada.io/name": "deploy-zp7hq-784b697bfc",
"work.karmada.io/namespace": "karmada-es-member2"
},
"labels": {
"karmada.io/managed": "true",
"propagationpolicy.karmada.io/permanent-id": "c0e99a2d-eab7-4e55-851b-8589e467103c",
"resourcebinding.karmada.io/permanent-id": "32c43d16-140d-48aa-9422-9aa3bd50b6a7",
"resourcetemplate.karmada.io/retain-replicas": "true",
"work.karmada.io/permanent-id": "8e251b6c-7e8a-4151-8808-7ba265b12fdf"
},
"name": "deploy-zp7hq",
"namespace": "karmadatest-5zbq7"
},
"spec": {
"progressDeadlineSeconds": 600,
"replicas": 5,
"revisionHistoryLimit": 10,
"selector": {
"matchLabels": {
"app": "nginx"
}
},
"strategy": {
"rollingUpdate": {
"maxSurge": "25%",
"maxUnavailable": "25%"
},
"type": "RollingUpdate"
},
"template": {
"metadata": {
"creationTimestamp": null,
"labels": {
"app": "nginx"
}
},
"spec": {
"containers": [
{
"image": "nginx:1.19.0",
"imagePullPolicy": "IfNotPresent",
"name": "nginx",
"ports": [
{
"containerPort": 80,
"protocol": "TCP"
}
],
"resources": {
"limits": {
"cpu": "100m"
},
"requests": {
"cpu": "10m"
}
},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File"
}
],
"dnsPolicy": "ClusterFirst",
"restartPolicy": "Always",
"schedulerName": "default-scheduler",
"securityContext": {},
"terminationGracePeriodSeconds": 30
}
}
}
}
]
}
},
"status": {
"conditions": [
{
"type": "Applied",
"status": "True",
"lastTransitionTime": "2024-07-27T09:51:59Z",
"reason": "AppliedSuccessful",
"message": "Manifest has been successfully applied"
}
],
"manifestStatuses": [
{
"identifier": {
"ordinal": 0,
"group": "apps",
"version": "v1",
"kind": "Deployment",
"resource": "",
"namespace": "karmadatest-5zbq7",
"name": "deploy-zp7hq"
},
"status": {
"availableReplicas": 3,
"generation": 5,
"observedGeneration": 5,
"readyReplicas": 3,
"replicas": 3,
"resourceTemplateGeneration": 5,
"updatedReplicas": 3
},
"health": "Healthy"
}
]
}
}
key log
2024-07-27T09:51:59.434966058Z stderr F I0727 09:51:59.434821 1 aggregatestatus.go:78] Grab deployment(karmadatest-5zbq7/deploy-zp7hq) status from cluster(member1), replicas: 3, ready: 3, updated: 3, available: 3, unavailable: 0
2024-07-27T09:51:59.43510654Z stderr F I0727 09:51:59.435048 1 aggregatestatus.go:78] Grab deployment(karmadatest-5zbq7/deploy-zp7hq) status from cluster(member2), replicas: 3, ready: 3, updated: 3, available: 3, unavailable: 0
2024-07-27T09:51:59.435246522Z stderr F I0727 09:51:59.435203 1 aggregatestatus.go:109] Ignore update deployment(karmadatest-5zbq7/deploy-zp7hq) status as up to date
2024-07-27T09:51:59.493172419Z stderr F I0727 09:51:59.493113 1 aggregatestatus.go:78] Grab deployment(karmadatest-5zbq7/deploy-zp7hq) status from cluster(member1), replicas: 3, ready: 3, updated: 3, available: 3, unavailable: 0
2024-07-27T09:51:59.493180754Z stderr F I0727 09:51:59.493137 1 aggregatestatus.go:78] Grab deployment(karmadatest-5zbq7/deploy-zp7hq) status from cluster(member2), replicas: 3, ready: 3, updated: 3, available: 3, unavailable: 0
2024-07-27T09:51:59.563255022Z stderr F I0727 09:51:59.563115 1 common.go:144] Update resource(apps/v1, Resource=deployments/karmadatest-5zbq7/deploy-zp7hq) status successfully.
2024-07-27T09:51:59.580637562Z stderr F I0727 09:51:59.580560 1 aggregatestatus.go:78] Grab deployment(karmadatest-5zbq7/deploy-zp7hq) status from cluster(member1), replicas: 3, ready: 3, updated: 3, available: 3, unavailable: 0
2024-07-27T09:51:59.586625873Z stderr F I0727 09:51:59.586527 1 execution_controller.go:141] Sync work(karmada-es-member2/deploy-zp7hq-784b697bfc) to cluster(member2) successful.
2024-07-27T09:51:59.588513499Z stderr F I0727 09:51:59.588291 1 common.go:144] Update resource(apps/v1, Resource=deployments/karmadatest-5zbq7/deploy-zp7hq) status successfully.
2024-07-27T09:51:59.598877556Z stderr F I0727 09:51:59.598800 1 aggregatestatus.go:78] Grab deployment(karmadatest-5zbq7/deploy-zp7hq) status from cluster(member1), replicas: 3, ready: 3, updated: 3, available: 3, unavailable: 0
2024-07-27T09:51:59.59888505Z stderr F I0727 09:51:59.598825 1 aggregatestatus.go:78] Grab deployment(karmadatest-5zbq7/deploy-zp7hq) status from cluster(member2), replicas: 3, ready: 3, updated: 3, available: 3, unavailable: 0
2024-07-27T09:51:59.598889969Z stderr F I0727 09:51:59.598832 1 aggregatestatus.go:109] Ignore update deployment(karmadatest-5zbq7/deploy-zp7hq) status as up to date
2024-07-27T09:51:59.598894838Z stderr F I0727 09:51:59.598843 1 common.go:128] Ignore update resource(apps/v1, Resource=deployments/karmadatest-5zbq7/deploy-zp7hq) status as up to date.
This case have two reason:
- also for some abnormal reason result in
binding.Status.AggregatedStatus[n].Statusbe nil.
https://github.com/karmada-io/karmada/blob/ec81ac7c80ef500b936e7f7d9239b4babf7de05a/pkg/resourceinterpreter/default/native/aggregatestatus.go#L70-L73
- following
oldStatusis from cache, and is out-dated. we mistakenly judged that an update could be omitted based on anoldStatus, but it should not be omitted, and once omitted, there is no chance to make up for the update.
https://github.com/karmada-io/karmada/blob/ec81ac7c80ef500b936e7f7d9239b4babf7de05a/pkg/resourceinterpreter/default/native/aggregatestatus.go#L103-L111
also for some abnormal reason result in binding.Status.AggregatedStatus[n].Status be nil. https://github.com/karmada-io/karmada/blob/ec81ac7c80ef500b936e7f7d9239b4babf7de05a/pkg/resourceinterpreter/default/native/aggregatestatus.go#L70-L73
I found why binding.Status.AggregatedStatus[n].Status can be nil:
Binding:
"aggregatedStatus": [{
"clusterName": "member1",
"appliedMessage": "Failed to apply all manifests (0/1): Operation cannot be fulfilled on deployments.apps \"deploy-9qm68\": the object has been modified; please apply your changes to the latest version and try again",
"health": "Unknown"
}, {
"clusterName": "member2",
"status": {
"availableReplicas": 3,
"generation": 8,
"observedGeneration": 8,
"readyReplicas": 3,
"replicas": 3,
"resourceTemplateGeneration": 7,
"updatedReplicas": 3
},
"applied": true,
"health": "Healthy"
}]
Work:
"status": {
"conditions": [{
"type": "Dispatching",
"status": "True",
"lastTransitionTime": "2024-08-19T08:22:14Z",
"reason": "Dispatching",
"message": "Work is being dispatched to member clusters."
}, {
"type": "Applied",
"status": "False",
"lastTransitionTime": "2024-08-19T08:22:40Z",
"reason": "AppliedFailed",
"message": "Failed to apply all manifests (0/1): Operation cannot be fulfilled on deployments.apps \"deploy-9qm68\": the object has been modified; please apply your changes to the latest version and try again"
}],
"manifestStatuses": [{
"identifier": {
"ordinal": 0,
"group": "apps",
"version": "v1",
"kind": "Deployment",
"resource": "",
"namespace": "karmadatest-wz755",
"name": "deploy-9qm68"
},
"status": {
"availableReplicas": 3,
"generation": 8,
"observedGeneration": 8,
"readyReplicas": 3,
"replicas": 3,
"resourceTemplateGeneration": 7,
"updatedReplicas": 3
},
"health": "Healthy"
}]
}
then deployment_replicas_syncer_controller can't finish its sync with following error:
deployment_replicas_syncer_controller.go:179] wait until aggregated status of cluster member1 collected
You can see the status of work/member1 appears normal, yet its condition/Applied is set to false as a result of conflicts during the cluster object update. Consequently, the rb_status_controller does not acknowledge the current status of the member cluster and declines to aggregate the status into the resource template.
However, an issue arises when the condition/Applied is introduced by the execution_controller during the update of the cluster object. Normally, if the update fails due to a conflict with the cluster object, it should trigger a retry of the reconciliation process. Yet, this does not appear to be happening as expected.
@RainbowMango: Closing this issue.
In response to this:
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
