pravega-operator icon indicating copy to clipboard operation
pravega-operator copied to clipboard

Published 20 hours ago • verified publisher icon pravega

Sporadic "failed to reconcile pravega cluster" messages

Open deenav opened this issue 5 years ago • 5 comments

Operator logs shows failed to reconcile pravega cluster (pravega): failed to sync cluster version: Operation cannot be fulfilled on pravegaclusters.pravega.pravega.io \"pravega\": the object has been modified; please apply your changes to the latest version and try again message immediate after upgrade completed.

With Pravega operator : 0.4.0-rc1 , I have tried upgrading Pravega version from 0.5.0-2236.5228e2d to 0.5.0-2236.5228e2d-upgrade and the upgrade completed successfully. But after upgrade completion, observing below log messages in pravega-operator logs.

time="2019-05-21T11:29:59Z" level=info msg="syncing to version '0.5.0-2236.5228e2d-upgrade' completed"
time="2019-05-21T11:29:59Z" level=info msg="Webhook is handling incoming requests"
time="2019-05-21T11:29:59Z" level=info msg="Webhook is handling incoming requests"
time="2019-05-21T11:29:59Z" level=info msg="failed to reconcile pravega cluster (pravega): failed to sync cluster version: Operation cannot be fulfilled on pravegaclusters.pravega.pravega.io \"pravega\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-05-21T11:30:00Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:30:01Z" level=info msg="syncing to version '0.5.0-2236.5228e2d-upgrade' completed"

Note: This message is observed only once out of 4 retries but filing this issue for tracking.

Complete Pravega-operator logs for the upgrade:

time="2019-05-21T11:04:11Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:04:30Z" level=info msg="Webhook is handling incoming requests"
time="2019-05-21T11:04:30Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:04:31Z" level=info msg="syncing cluster version from 0.5.0-2236.5228e2d to 0.5.0-2236.5228e2d-upgrade"
time="2019-05-21T11:04:31Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:04:31Z" level=info msg="updating statefulset (pravega-bookie) template image to 'deenadock/bookkeeper:0.5.0-2236.5228e2d-upgrade'"
time="2019-05-21T11:04:42Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:04:42Z" level=info msg="statefulset (pravega-bookie) status: 0 updated, 4 ready, 4 target"
time="2019-05-21T11:04:42Z" level=info msg="upgrading pod: pravega-bookie-0"
time="2019-05-21T11:05:12Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:05:12Z" level=info msg="statefulset (pravega-bookie) status: 0 updated, 4 ready, 4 target"
time="2019-05-21T11:05:12Z" level=info msg="upgrading pod: pravega-bookie-0"
time="2019-05-21T11:05:42Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:05:43Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:05:43Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:05:43Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:06:13Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:06:13Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:06:43Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:06:43Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:07:13Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:07:14Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:07:44Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:07:44Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:08:14Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:08:14Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:08:44Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:08:45Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:09:15Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:09:15Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:09:45Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:09:45Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:10:15Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:10:16Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:10:46Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:10:46Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:11:16Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:11:16Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:11:46Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:11:47Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:12:17Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:12:17Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:12:47Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:12:47Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:13:17Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:13:18Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:13:48Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:13:48Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:14:18Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:14:18Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:14:48Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:14:49Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:15:19Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:15:19Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:15:49Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:15:49Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:16:19Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:16:20Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:16:50Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:16:50Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:17:20Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:17:20Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:17:50Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:17:51Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:18:21Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:18:21Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:18:51Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:18:51Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:19:21Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:19:22Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:19:52Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:19:52Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:20:22Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:20:22Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:20:52Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:20:53Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:21:23Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:21:23Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:21:53Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:21:53Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 3 ready, 4 target"
time="2019-05-21T11:22:23Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:22:24Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 4 ready, 4 target"
time="2019-05-21T11:22:24Z" level=info msg="upgrading pod: pravega-bookie-1"
time="2019-05-21T11:22:24Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:22:24Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 4 ready, 4 target"
time="2019-05-21T11:22:24Z" level=info msg="upgrading pod: pravega-bookie-1"
time="2019-05-21T11:22:54Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:22:54Z" level=info msg="statefulset (pravega-bookie) status: 1 updated, 4 ready, 4 target"
time="2019-05-21T11:22:54Z" level=info msg="upgrading pod: pravega-bookie-1"
time="2019-05-21T11:23:24Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:23:24Z" level=info msg="statefulset (pravega-bookie) status: 2 updated, 3 ready, 4 target"
time="2019-05-21T11:23:24Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:23:25Z" level=info msg="statefulset (pravega-bookie) status: 2 updated, 3 ready, 4 target"
time="2019-05-21T11:23:54Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:23:55Z" level=info msg="statefulset (pravega-bookie) status: 2 updated, 4 ready, 4 target"
time="2019-05-21T11:23:55Z" level=info msg="upgrading pod: pravega-bookie-2"
time="2019-05-21T11:23:55Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:23:55Z" level=info msg="statefulset (pravega-bookie) status: 2 updated, 4 ready, 4 target"
time="2019-05-21T11:23:55Z" level=info msg="upgrading pod: pravega-bookie-2"
time="2019-05-21T11:24:25Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:24:25Z" level=info msg="statefulset (pravega-bookie) status: 2 updated, 4 ready, 4 target"
time="2019-05-21T11:24:25Z" level=info msg="upgrading pod: pravega-bookie-2"
time="2019-05-21T11:24:55Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:24:55Z" level=info msg="statefulset (pravega-bookie) status: 3 updated, 3 ready, 4 target"
time="2019-05-21T11:24:56Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:24:56Z" level=info msg="statefulset (pravega-bookie) status: 3 updated, 3 ready, 4 target"
time="2019-05-21T11:25:26Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:25:26Z" level=info msg="statefulset (pravega-bookie) status: 3 updated, 4 ready, 4 target"
time="2019-05-21T11:25:26Z" level=info msg="upgrading pod: pravega-bookie-3"
time="2019-05-21T11:25:26Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:25:26Z" level=info msg="statefulset (pravega-bookie) status: 3 updated, 4 ready, 4 target"
time="2019-05-21T11:25:26Z" level=info msg="upgrading pod: pravega-bookie-3"
time="2019-05-21T11:25:56Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:25:56Z" level=info msg="statefulset (pravega-bookie) status: 3 updated, 4 ready, 4 target"
time="2019-05-21T11:25:56Z" level=info msg="upgrading pod: pravega-bookie-3"
time="2019-05-21T11:26:26Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:26:27Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 3 ready, 4 target"
time="2019-05-21T11:26:27Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:26:27Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 3 ready, 4 target"
time="2019-05-21T11:26:57Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:26:57Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:26:57Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:26:57Z" level=info msg="updating statefulset (pravega-pravega-segmentstore) template image to 'deenadock/pravega:0.5.0-2236.5228e2d-upgrade'"
time="2019-05-21T11:26:57Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:26:57Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:26:57Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:26:57Z" level=info msg="statefulset (pravega-pravega-segmentstore) status: 0 updated, 3 ready, 3 target"
time="2019-05-21T11:27:27Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:27:27Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:27:27Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:27:27Z" level=info msg="statefulset (pravega-pravega-segmentstore) status: 1 updated, 2 ready, 3 target"
time="2019-05-21T11:27:27Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:27:28Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:27:28Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:27:28Z" level=info msg="statefulset (pravega-pravega-segmentstore) status: 1 updated, 2 ready, 3 target"
time="2019-05-21T11:27:57Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:27:58Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:27:58Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:27:58Z" level=info msg="statefulset (pravega-pravega-segmentstore) status: 1 updated, 2 ready, 3 target"
time="2019-05-21T11:27:58Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:27:58Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:27:58Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:27:58Z" level=info msg="statefulset (pravega-pravega-segmentstore) status: 1 updated, 2 ready, 3 target"
time="2019-05-21T11:28:28Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:28:28Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:28:28Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:28:28Z" level=info msg="statefulset (pravega-pravega-segmentstore) status: 2 updated, 2 ready, 3 target"
time="2019-05-21T11:28:58Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:28:58Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:28:58Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:28:58Z" level=info msg="statefulset (pravega-pravega-segmentstore) status: 3 updated, 2 ready, 3 target"
time="2019-05-21T11:28:58Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:28:59Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:28:59Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:28:59Z" level=info msg="statefulset (pravega-pravega-segmentstore) status: 3 updated, 2 ready, 3 target"
time="2019-05-21T11:29:28Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:29:29Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:29:29Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:29:29Z" level=info msg="statefulset (pravega-pravega-segmentstore) status: 3 updated, 3 ready, 3 target"
time="2019-05-21T11:29:29Z" level=info msg="segmentstore version sync has been completed"
time="2019-05-21T11:29:29Z" level=info msg="updating deployment (pravega-pravega-controller) pod template image to 'deenadock/pravega:0.5.0-2236.5228e2d-upgrade'"
time="2019-05-21T11:29:29Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:29:29Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:29:29Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:29:29Z" level=info msg="statefulset (pravega-pravega-segmentstore) status: 3 updated, 3 ready, 3 target"
time="2019-05-21T11:29:29Z" level=info msg="segmentstore version sync has been completed"
time="2019-05-21T11:29:29Z" level=info msg="deployment (pravega-pravega-controller) status: 1 updated, 2 ready, 3 target"
time="2019-05-21T11:29:29Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:29:30Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:29:30Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:29:30Z" level=info msg="statefulset (pravega-pravega-segmentstore) status: 3 updated, 3 ready, 3 target"
time="2019-05-21T11:29:30Z" level=info msg="segmentstore version sync has been completed"
time="2019-05-21T11:29:30Z" level=info msg="deployment (pravega-pravega-controller) status: 1 updated, 2 ready, 3 target"
time="2019-05-21T11:29:59Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:29:59Z" level=info msg="statefulset (pravega-bookie) status: 4 updated, 4 ready, 4 target"
time="2019-05-21T11:29:59Z" level=info msg="bookkeeper version sync has been completed"
time="2019-05-21T11:29:59Z" level=info msg="statefulset (pravega-pravega-segmentstore) status: 3 updated, 3 ready, 3 target"
time="2019-05-21T11:29:59Z" level=info msg="segmentstore version sync has been completed"
time="2019-05-21T11:29:59Z" level=info msg="deployment (pravega-pravega-controller) status: 2 updated, 2 ready, 2 target"
time="2019-05-21T11:29:59Z" level=info msg="controller version sync has been completed"
time="2019-05-21T11:29:59Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:29:59Z" level=info msg="syncing to version '0.5.0-2236.5228e2d-upgrade' completed"
time="2019-05-21T11:29:59Z" level=info msg="Webhook is handling incoming requests"
time="2019-05-21T11:29:59Z" level=info msg="Webhook is handling incoming requests"
time="2019-05-21T11:29:59Z" level=info msg="failed to reconcile pravega cluster (pravega): failed to sync cluster version: Operation cannot be fulfilled on pravegaclusters.pravega.pravega.io \"pravega\": the object has been modified; please apply your changes to the latest version and try again"
time="2019-05-21T11:30:00Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:30:01Z" level=info msg="syncing to version '0.5.0-2236.5228e2d-upgrade' completed"
time="2019-05-21T11:30:01Z" level=info msg="Webhook is handling incoming requests"
time="2019-05-21T11:30:01Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:30:31Z" level=info msg="Reconciling PravegaCluster default/pravega\n"
time="2019-05-21T11:31:01Z" level=info msg="Reconciling PravegaCluster default/pravega\n"

deenav avatar May 21 '19 12:05 deenav

This is because an update happens before another update in one reconciling loop so the later update is using an outdated resourceVersion. This is usually a benign error and it can be fixed by entering the next reconciling loop. I'm tracking down what exactly happened.

Tristan1900 avatar May 22 '19 05:05 Tristan1900

After some investigations and experiments, I suppose the following lead to this error. At the beginning of reconciling loop, the operator sdk uses a GET method to fetch the latest Pravega object and use it for the following updates. According to the doc, the default behavior of GET is to get from a local cache and that cache has only eventual consistency with the apiserver. So what happens is that there are two UPDATE here and here that are executed almost at the same time. The apiserver gets two updates but the local cache somehow only get synced with the first update at the start of the next reconciling loop, which means the next loop is using a stale Pravega object and leads to this error. I have noticed that the return of GET is updated after a short period of time. We can reproduce this inconsistency between the result of GET and UPDATE easily by a pseudo code like this:

for i := 0; i < 10; i++{
  UPDATE(i)
  n := GET()
  print(n)
}

As in my last comment, this is a benign error and GET will eventually fetch the latest Pravega object. Do you think we should close it or somehow fix it? @adrianmo Thank you!

Tristan1900 avatar May 23 '19 03:05 Tristan1900

This is a benign error, it can be fixed without using cache but that will reduce the performance of the operator and it is suggested to use this cache by the operator framework, see the last line of this. Can we close it or otherwise? Thanks @deenav

Tristan1900 avatar Jul 17 '19 18:07 Tristan1900

Downgrading to P3 since there is no functionality loss here.

pbelgundi avatar Sep 03 '19 06:09 pbelgundi

@SrishT, from the error message and analysis presented by @Tristan1900, this does not seem to be an "upgrade" issue but rather an issue with the k8s client we use and related caching. Could you please try to see if this error is reproducible without doing an upgrade, by changing the number of replicas for segment store after the cluster is deployed?

pbelgundi avatar Oct 11 '19 07:10 pbelgundi