bottlerocket-update-operator
bottlerocket-update-operator copied to clipboard
Published
20 hours ago •
bottlerocket-os
bottlerocket-os
Failed to convert from brupop.bottlerocket.aws/v2 to brupop.bottlerocket.aws/v1 version
Image I'm using:
brupop: public.ecr.aws/bottlerocket/bottlerocket-update-operator:v0.2.2
EKS:
Server Version: version.Info{Major:"1", Minor:"23+", GitVersion:"v1.23.7-eks-4721010", GitCommit:"b77d9473a02fbfa834afa67d677fd12d690b195f", GitTreeState:"clean", BuildDate:"2022-06-27T22:19:07Z", GoVersion:"go1.17.10", Compiler:"gc", Platform:"linux/amd64"}
Issue or Feature Request:
Every second, the /crdconvert endpoint of the webhook is called by the Kubernetes API server, and the conversion fails with the following message:
Failed to convert from brupop.bottlerocket.aws/v2 to brupop.bottlerocket.aws/v1 version
I've attached the logs related to one of those requests.
{"v":0,"name":"apiserver","msg":"[HTTP REQUEST - START]","level":30,"hostname":"brupop-apiserver-8575cc4658-4mm6p","pid":1,"time":"2022-09-21T02:26:10.547533378+00:00","target":"apiserver::telemetry","line":42,"file":"apiserver/src/telemetry.rs","http.method":"POST","request_id":"6b8ff3f7-85eb-427a-832e-472d77d0c339","http.flavor":"1.1","http.target":"/crdconvert?timeout=30s","otel.kind":"server","http.user_agent":"kube-apiserver-admission","http.host":"brupop-apiserver.brupop-bottlerocket-aws.svc:443","http.route":"/crdconvert","http.scheme":"https","http.client_ip":"10.10.1.106:41894"}
{"v":0,"name":"apiserver","msg":"[HTTP REQUEST - EVENT] Original conversion request","level":30,"hostname":"brupop-apiserver-8575cc4658-4mm6p","pid":1,"time":"2022-09-21T02:26:10.547772320+00:00","target":"apiserver::api::node","line":51,"file":"apiserver/src/api/node.rs","conversion_req":"Json(ConversionRequest { kind: \"ConversionReview\", api_version: \"apiextensions.k8s.io/v1\", request: Request { uid: \"f238a9a2-1f46-4cce-887d-e4459bb096a1\", desired_api_version: \"brupop.bottlerocket.aws/v1\", objects: [Object({\"apiVersion\": String(\"brupop.bottlerocket.aws/v2\"), \"kind\": String(\"BottlerocketShadow\"), \"metadata\": Object({\"creationTimestamp\": String(\"2022-09-18T15:59:10Z\"), \"generation\": Number(1), \"managedFields\": Array([Object({\"apiVersion\": String(\"brupop.bottlerocket.aws/v2\"), \"fieldsType\": String(\"FieldsV1\"), \"fieldsV1\": Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"5bd7850b-63f7-4307-b317-fd32c12bfd7d\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}), \"manager\": String(\"unknown\"), \"operation\": String(\"Update\"), \"time\": String(\"2022-09-18T15:59:10Z\")}), Object({\"apiVersion\": String(\"brupop.bottlerocket.aws/v2\"), \"fieldsType\": String(\"FieldsV1\"), \"fieldsV1\": Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}), \"manager\": String(\"unknown\"), \"operation\": String(\"Update\"), \"subresource\": String(\"status\"), \"time\": String(\"2022-09-18T15:59:30Z\")})]), \"name\": String(\"brs-ip-10-10-110-247.ap-northeast-1.compute.internal\"), \"namespace\": String(\"brupop-bottlerocket-aws\"), \"ownerReferences\": Array([Object({\"apiVersion\": String(\"v1\"), \"kind\": String(\"Node\"), \"name\": String(\"ip-10-10-110-247.ap-northeast-1.compute.internal\"), \"uid\": String(\"5bd7850b-63f7-4307-b317-fd32c12bfd7d\")})]), \"uid\": String(\"7f8c24a2-08c9-40d3-8d54-a2d38093de29\")}), \"spec\": Object({\"state\": String(\"Idle\"), \"state_transition_timestamp\": Null, \"version\": Null}), \"status\": Object({\"crash_count\": Number(0), \"current_state\": String(\"Idle\"), \"current_version\": String(\"1.9.2\"), \"target_version\": String(\"1.9.2\")})})] } })","http.method":"POST","request_id":"6b8ff3f7-85eb-427a-832e-472d77d0c339","http.flavor":"1.1","http.target":"/crdconvert?timeout=30s","otel.kind":"server","http.user_agent":"kube-apiserver-admission","http.host":"brupop-apiserver.brupop-bottlerocket-aws.svc:443","http.route":"/crdconvert","http.scheme":"https","http.client_ip":"10.10.1.106:41894"}
{"v":0,"name":"apiserver","msg":"[CONVERT_OBJECTS - START]","level":30,"hostname":"brupop-apiserver-8575cc4658-4mm6p","pid":1,"time":"2022-09-21T02:26:10.547884152+00:00","target":"apiserver::webhook","line":131,"file":"apiserver/src/webhook/mod.rs","objects":"[Object({\"apiVersion\": String(\"brupop.bottlerocket.aws/v2\"), \"kind\": String(\"BottlerocketShadow\"), \"metadata\": Object({\"creationTimestamp\": String(\"2022-09-18T15:59:10Z\"), \"generation\": Number(1), \"managedFields\": Array([Object({\"apiVersion\": String(\"brupop.bottlerocket.aws/v2\"), \"fieldsType\": String(\"FieldsV1\"), \"fieldsV1\": Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"5bd7850b-63f7-4307-b317-fd32c12bfd7d\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}), \"manager\": String(\"unknown\"), \"operation\": String(\"Update\"), \"time\": String(\"2022-09-18T15:59:10Z\")}), Object({\"apiVersion\": String(\"brupop.bottlerocket.aws/v2\"), \"fieldsType\": String(\"FieldsV1\"), \"fieldsV1\": Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}), \"manager\": String(\"unknown\"), \"operation\": String(\"Update\"), \"subresource\": String(\"status\"), \"time\": String(\"2022-09-18T15:59:30Z\")})]), \"name\": String(\"brs-ip-10-10-110-247.ap-northeast-1.compute.internal\"), \"namespace\": String(\"brupop-bottlerocket-aws\"), \"ownerReferences\": Array([Object({\"apiVersion\": String(\"v1\"), \"kind\": String(\"Node\"), \"name\": String(\"ip-10-10-110-247.ap-northeast-1.compute.internal\"), \"uid\": String(\"5bd7850b-63f7-4307-b317-fd32c12bfd7d\")})]), \"uid\": String(\"7f8c24a2-08c9-40d3-8d54-a2d38093de29\")}), \"spec\": Object({\"state\": String(\"Idle\"), \"state_transition_timestamp\": Null, \"version\": Null}), \"status\": Object({\"crash_count\": Number(0), \"current_state\": String(\"Idle\"), \"current_version\": String(\"1.9.2\"), \"target_version\": String(\"1.9.2\")})})]","http.method":"POST","request_id":"6b8ff3f7-85eb-427a-832e-472d77d0c339","http.flavor":"1.1","http.target":"/crdconvert?timeout=30s","otel.kind":"server","http.user_agent":"kube-apiserver-admission","http.host":"brupop-apiserver.brupop-bottlerocket-aws.svc:443","http.route":"/crdconvert","http.scheme":"https","http.client_ip":"10.10.1.106:41894","desired_version":"\"brupop.bottlerocket.aws/v1\""}
{"v":0,"name":"apiserver","msg":"[CHAINED_CONVERT_OBJECT - START]","level":30,"hostname":"brupop-apiserver-8575cc4658-4mm6p","pid":1,"time":"2022-09-21T02:26:10.547937962+00:00","target":"apiserver::webhook","line":187,"file":"apiserver/src/webhook/mod.rs","objects":"[Object({\"apiVersion\": String(\"brupop.bottlerocket.aws/v2\"), \"kind\": String(\"BottlerocketShadow\"), \"metadata\": Object({\"creationTimestamp\": String(\"2022-09-18T15:59:10Z\"), \"generation\": Number(1), \"managedFields\": Array([Object({\"apiVersion\": String(\"brupop.bottlerocket.aws/v2\"), \"fieldsType\": String(\"FieldsV1\"), \"fieldsV1\": Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"5bd7850b-63f7-4307-b317-fd32c12bfd7d\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}), \"manager\": String(\"unknown\"), \"operation\": String(\"Update\"), \"time\": String(\"2022-09-18T15:59:10Z\")}), Object({\"apiVersion\": String(\"brupop.bottlerocket.aws/v2\"), \"fieldsType\": String(\"FieldsV1\"), \"fieldsV1\": Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}), \"manager\": String(\"unknown\"), \"operation\": String(\"Update\"), \"subresource\": String(\"status\"), \"time\": String(\"2022-09-18T15:59:30Z\")})]), \"name\": String(\"brs-ip-10-10-110-247.ap-northeast-1.compute.internal\"), \"namespace\": String(\"brupop-bottlerocket-aws\"), \"ownerReferences\": Array([Object({\"apiVersion\": String(\"v1\"), \"kind\": String(\"Node\"), \"name\": String(\"ip-10-10-110-247.ap-northeast-1.compute.internal\"), \"uid\": String(\"5bd7850b-63f7-4307-b317-fd32c12bfd7d\")})]), \"uid\": String(\"7f8c24a2-08c9-40d3-8d54-a2d38093de29\")}), \"spec\": Object({\"state\": String(\"Idle\"), \"state_transition_timestamp\": Null, \"version\": Null}), \"status\": Object({\"crash_count\": Number(0), \"current_state\": String(\"Idle\"), \"current_version\": String(\"1.9.2\"), \"target_version\": String(\"1.9.2\")})})]","http.method":"POST","request_id":"6b8ff3f7-85eb-427a-832e-472d77d0c339","http.flavor":"1.1","http.target":"/crdconvert?timeout=30s","otel.kind":"server","http.user_agent":"kube-apiserver-admission","http.host":"brupop-apiserver.brupop-bottlerocket-aws.svc:443","http.route":"/crdconvert","http.scheme":"https","http.client_ip":"10.10.1.106:41894","desired_version":"\"brupop.bottlerocket.aws/v1\""}
{"v":0,"name":"apiserver","msg":"[CHAINED_CONVERT_OBJECT - EVENT] apiserver::webhook","level":50,"hostname":"brupop-apiserver-8575cc4658-4mm6p","pid":1,"time":"2022-09-21T02:26:10.547976723+00:00","target":"apiserver::webhook","line":188,"file":"apiserver/src/webhook/mod.rs","error":"Failed to convert from brupop.bottlerocket.aws/v2 to brupop.bottlerocket.aws/v1 version","objects":"[Object({\"apiVersion\": String(\"brupop.bottlerocket.aws/v2\"), \"kind\": String(\"BottlerocketShadow\"), \"metadata\": Object({\"creationTimestamp\": String(\"2022-09-18T15:59:10Z\"), \"generation\": Number(1), \"managedFields\": Array([Object({\"apiVersion\": String(\"brupop.bottlerocket.aws/v2\"), \"fieldsType\": String(\"FieldsV1\"), \"fieldsV1\": Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"5bd7850b-63f7-4307-b317-fd32c12bfd7d\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}), \"manager\": String(\"unknown\"), \"operation\": String(\"Update\"), \"time\": String(\"2022-09-18T15:59:10Z\")}), Object({\"apiVersion\": String(\"brupop.bottlerocket.aws/v2\"), \"fieldsType\": String(\"FieldsV1\"), \"fieldsV1\": Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}), \"manager\": String(\"unknown\"), \"operation\": String(\"Update\"), \"subresource\": String(\"status\"), \"time\": String(\"2022-09-18T15:59:30Z\")})]), \"name\": String(\"brs-ip-10-10-110-247.ap-northeast-1.compute.internal\"), \"namespace\": String(\"brupop-bottlerocket-aws\"), \"ownerReferences\": Array([Object({\"apiVersion\": String(\"v1\"), \"kind\": String(\"Node\"), \"name\": String(\"ip-10-10-110-247.ap-northeast-1.compute.internal\"), \"uid\": String(\"5bd7850b-63f7-4307-b317-fd32c12bfd7d\")})]), \"uid\": String(\"7f8c24a2-08c9-40d3-8d54-a2d38093de29\")}), \"spec\": Object({\"state\": String(\"Idle\"), \"state_transition_timestamp\": Null, \"version\": Null}), \"status\": Object({\"crash_count\": Number(0), \"current_state\": String(\"Idle\"), \"current_version\": String(\"1.9.2\"), \"target_version\": String(\"1.9.2\")})})]","http.method":"POST","request_id":"6b8ff3f7-85eb-427a-832e-472d77d0c339","http.flavor":"1.1","http.target":"/crdconvert?timeout=30s","otel.kind":"server","http.user_agent":"kube-apiserver-admission","http.host":"brupop-apiserver.brupop-bottlerocket-aws.svc:443","http.route":"/crdconvert","http.scheme":"https","http.client_ip":"10.10.1.106:41894","desired_version":"\"brupop.bottlerocket.aws/v1\""}
{"v":0,"name":"apiserver","msg":"[CHAINED_CONVERT_OBJECT - END]","level":30,"hostname":"brupop-apiserver-8575cc4658-4mm6p","pid":1,"time":"2022-09-21T02:26:10.548010613+00:00","target":"apiserver::webhook","line":187,"file":"apiserver/src/webhook/mod.rs","objects":"[Object({\"apiVersion\": String(\"brupop.bottlerocket.aws/v2\"), \"kind\": String(\"BottlerocketShadow\"), \"metadata\": Object({\"creationTimestamp\": String(\"2022-09-18T15:59:10Z\"), \"generation\": Number(1), \"managedFields\": Array([Object({\"apiVersion\": String(\"brupop.bottlerocket.aws/v2\"), \"fieldsType\": String(\"FieldsV1\"), \"fieldsV1\": Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"5bd7850b-63f7-4307-b317-fd32c12bfd7d\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}), \"manager\": String(\"unknown\"), \"operation\": String(\"Update\"), \"time\": String(\"2022-09-18T15:59:10Z\")}), Object({\"apiVersion\": String(\"brupop.bottlerocket.aws/v2\"), \"fieldsType\": String(\"FieldsV1\"), \"fieldsV1\": Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}), \"manager\": String(\"unknown\"), \"operation\": String(\"Update\"), \"subresource\": String(\"status\"), \"time\": String(\"2022-09-18T15:59:30Z\")})]), \"name\": String(\"brs-ip-10-10-110-247.ap-northeast-1.compute.internal\"), \"namespace\": String(\"brupop-bottlerocket-aws\"), \"ownerReferences\": Array([Object({\"apiVersion\": String(\"v1\"), \"kind\": String(\"Node\"), \"name\": String(\"ip-10-10-110-247.ap-northeast-1.compute.internal\"), \"uid\": String(\"5bd7850b-63f7-4307-b317-fd32c12bfd7d\")})]), \"uid\": String(\"7f8c24a2-08c9-40d3-8d54-a2d38093de29\")}), \"spec\": Object({\"state\": String(\"Idle\"), \"state_transition_timestamp\": Null, \"version\": Null}), \"status\": Object({\"crash_count\": Number(0), \"current_state\": String(\"Idle\"), \"current_version\": String(\"1.9.2\"), \"target_version\": String(\"1.9.2\")})})]","http.method":"POST","request_id":"6b8ff3f7-85eb-427a-832e-472d77d0c339","http.flavor":"1.1","http.target":"/crdconvert?timeout=30s","otel.kind":"server","http.user_agent":"kube-apiserver-admission","elapsed_milliseconds":0,"http.host":"brupop-apiserver.brupop-bottlerocket-aws.svc:443","http.route":"/crdconvert","http.scheme":"https","http.client_ip":"10.10.1.106:41894","desired_version":"\"brupop.bottlerocket.aws/v1\""}
{"v":0,"name":"apiserver","msg":"[CONVERT_OBJECTS - EVENT] apiserver::webhook","level":50,"hostname":"brupop-apiserver-8575cc4658-4mm6p","pid":1,"time":"2022-09-21T02:26:10.548041264+00:00","target":"apiserver::webhook","line":135,"file":"apiserver/src/webhook/mod.rs","error":"Failed to convert from brupop.bottlerocket.aws/v2 to brupop.bottlerocket.aws/v1 version","objects":"[Object({\"apiVersion\": String(\"brupop.bottlerocket.aws/v2\"), \"kind\": String(\"BottlerocketShadow\"), \"metadata\": Object({\"creationTimestamp\": String(\"2022-09-18T15:59:10Z\"), \"generation\": Number(1), \"managedFields\": Array([Object({\"apiVersion\": String(\"brupop.bottlerocket.aws/v2\"), \"fieldsType\": String(\"FieldsV1\"), \"fieldsV1\": Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"5bd7850b-63f7-4307-b317-fd32c12bfd7d\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}), \"manager\": String(\"unknown\"), \"operation\": String(\"Update\"), \"time\": String(\"2022-09-18T15:59:10Z\")}), Object({\"apiVersion\": String(\"brupop.bottlerocket.aws/v2\"), \"fieldsType\": String(\"FieldsV1\"), \"fieldsV1\": Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}), \"manager\": String(\"unknown\"), \"operation\": String(\"Update\"), \"subresource\": String(\"status\"), \"time\": String(\"2022-09-18T15:59:30Z\")})]), \"name\": String(\"brs-ip-10-10-110-247.ap-northeast-1.compute.internal\"), \"namespace\": String(\"brupop-bottlerocket-aws\"), \"ownerReferences\": Array([Object({\"apiVersion\": String(\"v1\"), \"kind\": String(\"Node\"), \"name\": String(\"ip-10-10-110-247.ap-northeast-1.compute.internal\"), \"uid\": String(\"5bd7850b-63f7-4307-b317-fd32c12bfd7d\")})]), \"uid\": String(\"7f8c24a2-08c9-40d3-8d54-a2d38093de29\")}), \"spec\": Object({\"state\": String(\"Idle\"), \"state_transition_timestamp\": Null, \"version\": Null}), \"status\": Object({\"crash_count\": Number(0), \"current_state\": String(\"Idle\"), \"current_version\": String(\"1.9.2\"), \"target_version\": String(\"1.9.2\")})})]","http.method":"POST","request_id":"6b8ff3f7-85eb-427a-832e-472d77d0c339","http.flavor":"1.1","http.target":"/crdconvert?timeout=30s","otel.kind":"server","http.user_agent":"kube-apiserver-admission","http.host":"brupop-apiserver.brupop-bottlerocket-aws.svc:443","http.route":"/crdconvert","http.scheme":"https","http.client_ip":"10.10.1.106:41894","desired_version":"\"brupop.bottlerocket.aws/v1\""}
{"v":0,"name":"apiserver","msg":"[CONVERT_OBJECTS - END]","level":30,"hostname":"brupop-apiserver-8575cc4658-4mm6p","pid":1,"time":"2022-09-21T02:26:10.548079624+00:00","target":"apiserver::webhook","line":131,"file":"apiserver/src/webhook/mod.rs","objects":"[Object({\"apiVersion\": String(\"brupop.bottlerocket.aws/v2\"), \"kind\": String(\"BottlerocketShadow\"), \"metadata\": Object({\"creationTimestamp\": String(\"2022-09-18T15:59:10Z\"), \"generation\": Number(1), \"managedFields\": Array([Object({\"apiVersion\": String(\"brupop.bottlerocket.aws/v2\"), \"fieldsType\": String(\"FieldsV1\"), \"fieldsV1\": Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"5bd7850b-63f7-4307-b317-fd32c12bfd7d\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}), \"manager\": String(\"unknown\"), \"operation\": String(\"Update\"), \"time\": String(\"2022-09-18T15:59:10Z\")}), Object({\"apiVersion\": String(\"brupop.bottlerocket.aws/v2\"), \"fieldsType\": String(\"FieldsV1\"), \"fieldsV1\": Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}), \"manager\": String(\"unknown\"), \"operation\": String(\"Update\"), \"subresource\": String(\"status\"), \"time\": String(\"2022-09-18T15:59:30Z\")})]), \"name\": String(\"brs-ip-10-10-110-247.ap-northeast-1.compute.internal\"), \"namespace\": String(\"brupop-bottlerocket-aws\"), \"ownerReferences\": Array([Object({\"apiVersion\": String(\"v1\"), \"kind\": String(\"Node\"), \"name\": String(\"ip-10-10-110-247.ap-northeast-1.compute.internal\"), \"uid\": String(\"5bd7850b-63f7-4307-b317-fd32c12bfd7d\")})]), \"uid\": String(\"7f8c24a2-08c9-40d3-8d54-a2d38093de29\")}), \"spec\": Object({\"state\": String(\"Idle\"), \"state_transition_timestamp\": Null, \"version\": Null}), \"status\": Object({\"crash_count\": Number(0), \"current_state\": String(\"Idle\"), \"current_version\": String(\"1.9.2\"), \"target_version\": String(\"1.9.2\")})})]","http.method":"POST","request_id":"6b8ff3f7-85eb-427a-832e-472d77d0c339","http.flavor":"1.1","http.target":"/crdconvert?timeout=30s","otel.kind":"server","http.user_agent":"kube-apiserver-admission","elapsed_milliseconds":0,"http.host":"brupop-apiserver.brupop-bottlerocket-aws.svc:443","http.route":"/crdconvert","http.scheme":"https","http.client_ip":"10.10.1.106:41894","desired_version":"\"brupop.bottlerocket.aws/v1\""}
{"v":0,"name":"apiserver","msg":"[HTTP REQUEST - EVENT] Converted response:","level":30,"hostname":"brupop-apiserver-8575cc4658-4mm6p","pid":1,"time":"2022-09-21T02:26:10.548120515+00:00","target":"apiserver::api::node","line":54,"file":"apiserver/src/api/node.rs","response_string":"\"{\\\"kind\\\":\\\"ConversionReview\\\",\\\"apiVersion\\\":\\\"apiextensions.k8s.io/v1\\\",\\\"response\\\":{\\\"uid\\\":\\\"f238a9a2-1f46-4cce-887d-e4459bb096a1\\\",\\\"result\\\":{\\\"status\\\":\\\"Failed\\\",\\\"message\\\":\\\"Failed to convert from brupop.bottlerocket.aws/v2 to brupop.bottlerocket.aws/v1 version\\\"},\\\"convertedObjects\\\":null}}\"","http.method":"POST","request_id":"6b8ff3f7-85eb-427a-832e-472d77d0c339","http.flavor":"1.1","http.target":"/crdconvert?timeout=30s","otel.kind":"server","http.user_agent":"kube-apiserver-admission","http.host":"brupop-apiserver.brupop-bottlerocket-aws.svc:443","http.route":"/crdconvert","http.scheme":"https","http.client_ip":"10.10.1.106:41894"}
{"v":0,"name":"apiserver","msg":"[HTTP REQUEST - END]","level":30,"hostname":"brupop-apiserver-8575cc4658-4mm6p","pid":1,"time":"2022-09-21T02:26:10.548198515+00:00","target":"apiserver::telemetry","line":42,"file":"apiserver/src/telemetry.rs","http.method":"POST","request_id":"6b8ff3f7-85eb-427a-832e-472d77d0c339","http.flavor":"1.1","http.target":"/crdconvert?timeout=30s","otel.kind":"server","http.user_agent":"kube-apiserver-admission","http.status_code":200,"otel.status_code":"OK","http.host":"brupop-apiserver.brupop-bottlerocket-aws.svc:443","http.route":"/crdconvert","http.scheme":"https","http.client_ip":"10.10.1.106:41894","elapsed_milliseconds":0}
Looking at the ConversionReview object, it seems the Kubernetes API server is trying to convert the BottlerocketShadow from brupop.bottlerocket.aws/v2 to brupop.bottlerocket.aws/v1.
ConversionRequest {
kind: "ConversionReview",
api_version: "apiextensions.k8s.io/v1",
request: Request {
uid: "f238a9a2-1f46-4cce-887d-e4459bb096a1",
desired_api_version: "brupop.bottlerocket.aws/v1",
objects: [
Object({
"apiVersion": String("brupop.bottlerocket.aws/v2"),
"kind": String("BottlerocketShadow"),
"metadata": Object({
"creationTimestamp": String("2022-09-18T15:59:10Z"),
"generation": Number(1),
"managedFields": Array([
Object({
"apiVersion": String("brupop.bottlerocket.aws/v2"),
"fieldsType": String("FieldsV1"),
"fieldsV1": Object({"f:metadata": Object({"f:ownerReferences": Object({".": Object({}), "k:{\"uid\":\"5bd7850b-63f7-4307-b317-fd32c12bfd7d\"}": Object({})})}), "f:spec": Object({".": Object({}), "f:state": Object({}), "f:state_transition_timestamp": Object({}), "f:version": Object({})})}),
"manager": String("unknown"),
"operation": String("Update"),
"time": String("2022-09-18T15:59:10Z")
}),
Object({
"apiVersion": String("brupop.bottlerocket.aws/v2"),
"fieldsType": String("FieldsV1"),
"fieldsV1": Object({"f:status": Object({".": Object({}), "f:crash_count": Object({}), "f:current_state": Object({}), "f:current_version": Object({}), "f:target_version": Object({})})}),
"manager": String("unknown"),
"operation": String("Update"),
"subresource": String("status"),
"time": String("2022-09-18T15:59:30Z")
})
]),
"name": String("brs-ip-10-10-110-247.ap-northeast-1.compute.internal"),
"namespace": String("brupop-bottlerocket-aws"),
"ownerReferences": Array([
Object({
"apiVersion": String("v1"),
"kind": String("Node"),
"name": String("ip-10-10-110-247.ap-northeast-1.compute.internal"),
"uid": String("5bd7850b-63f7-4307-b317-fd32c12bfd7d")
})
]),
"uid": String("7f8c24a2-08c9-40d3-8d54-a2d38093de29")
}),
"spec": Object({"state": String("Idle"), "state_transition_timestamp": Null, "version": Null}),
"status": Object({
"crash_count": Number(0),
"current_state": String("Idle"),
"current_version": String("1.9.2"),
"target_version": String("1.9.2")})
})
]
}
}
I've tried to update conversionReviewVersions in the bottlerocketshadows.brupop.bottlerocket.aws CRD to have v2 before v1, but it didn't change anything.
API servers send the first
ConversionReviewversion in theconversionReviewVersionslist they support.
https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/#webhook-request-and-response
I'm not sure what is the expected behavior from Kubernetes API server, but it should not be trying to convert from v2 to v1. And the case is anyway not handled in the webhook.
It is related to the changes introduced by https://github.com/bottlerocket-os/bottlerocket-update-operator/pull/221
The expected behavior is it should not be trying to convert from v2 to v1. I wondering if some of your pods are still in 0.2.1.
Would you mind share the result from kubectl describe pods -n brupop-bottlerocket-aws.
Thank you for your quick reply.
Here is the result of kubectl describe pods -n brupop-bottlerocket-aws
Name: brupop-agent-26xzj
Namespace: brupop-bottlerocket-aws
Priority: 500000
Priority Class Name: default
Service Account: brupop-agent-service-account
Node: ip-10-10-152-12.ap-northeast-1.compute.internal/10.10.152.12
Start Time: Wed, 21 Sep 2022 10:35:59 +0900
Labels: brupop.bottlerocket.aws/component=agent
controller-revision-hash=6fc8c5cbb5
pod-template-generation=4
Annotations: kubernetes.io/psp: eks.privileged
Status: Running
IP: 10.10.131.106
IPs:
IP: 10.10.131.106
Controlled By: DaemonSet/brupop-agent
Containers:
brupop:
Container ID: containerd://3b28aafad535536aba70e5dfcabddf3014232de938eedca184ec07b2e9895d75
Image: public.ecr.aws/bottlerocket/bottlerocket-update-operator:v0.2.2
Image ID: public.ecr.aws/bottlerocket/bottlerocket-update-operator@sha256:a6de31e1b3553e0c5b5401ec7f7cc435c150481f5c4827c061e523106b9748c0
Port: <none>
Host Port: <none>
Command:
./agent
State: Running
Started: Wed, 21 Sep 2022 10:36:06 +0900
Ready: True
Restart Count: 0
Limits:
memory: 50Mi
Requests:
cpu: 10m
memory: 50Mi
Environment:
RUST_LOG: info,agent::apiclient=warn,agent::agentclient=warn
MY_NODE_NAME: (v1:spec.nodeName)
Mounts:
/bin/apiclient from bottlerocket-apiclient (rw)
/etc/brupop-tls-keys from bottlerocket-tls-keys (rw)
/run/api.sock from bottlerocket-api-socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-swbq4 (ro)
/var/run/secrets/tokens/ from bottlerocket-agent-service-account-token (rw)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
bottlerocket-api-socket:
Type: HostPath (bare host directory volume)
Path: /run/api.sock
HostPathType: Socket
bottlerocket-apiclient:
Type: HostPath (bare host directory volume)
Path: /bin/apiclient
HostPathType: File
bottlerocket-agent-service-account-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3600
bottlerocket-tls-keys:
Type: Secret (a volume populated by a Secret)
SecretName: brupop-tls
Optional: false
kube-api-access-swbq4:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events: <none>
Name: brupop-agent-7b2wt
Namespace: brupop-bottlerocket-aws
Priority: 500000
Priority Class Name: default
Service Account: brupop-agent-service-account
Node: ip-10-10-36-216.ap-northeast-1.compute.internal/10.10.36.216
Start Time: Wed, 21 Sep 2022 09:29:20 +0900
Labels: brupop.bottlerocket.aws/component=agent
controller-revision-hash=6fc8c5cbb5
pod-template-generation=4
Annotations: kubernetes.io/psp: eks.privileged
Status: Running
IP: 10.10.38.180
IPs:
IP: 10.10.38.180
Controlled By: DaemonSet/brupop-agent
Containers:
brupop:
Container ID: containerd://15c612052cf84cba0a3f3df33ebaa37a56355b15574f289c2d191f9dd53627af
Image: public.ecr.aws/bottlerocket/bottlerocket-update-operator:v0.2.2
Image ID: public.ecr.aws/bottlerocket/bottlerocket-update-operator@sha256:a6de31e1b3553e0c5b5401ec7f7cc435c150481f5c4827c061e523106b9748c0
Port: <none>
Host Port: <none>
Command:
./agent
State: Running
Started: Wed, 21 Sep 2022 09:29:27 +0900
Ready: True
Restart Count: 0
Limits:
memory: 50Mi
Requests:
cpu: 10m
memory: 50Mi
Environment:
RUST_LOG: info,agent::apiclient=warn,agent::agentclient=warn
MY_NODE_NAME: (v1:spec.nodeName)
Mounts:
/bin/apiclient from bottlerocket-apiclient (rw)
/etc/brupop-tls-keys from bottlerocket-tls-keys (rw)
/run/api.sock from bottlerocket-api-socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qhcvq (ro)
/var/run/secrets/tokens/ from bottlerocket-agent-service-account-token (rw)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
bottlerocket-api-socket:
Type: HostPath (bare host directory volume)
Path: /run/api.sock
HostPathType: Socket
bottlerocket-apiclient:
Type: HostPath (bare host directory volume)
Path: /bin/apiclient
HostPathType: File
bottlerocket-agent-service-account-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3600
bottlerocket-tls-keys:
Type: Secret (a volume populated by a Secret)
SecretName: brupop-tls
Optional: false
kube-api-access-qhcvq:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events: <none>
Name: brupop-agent-7g44p
Namespace: brupop-bottlerocket-aws
Priority: 500000
Priority Class Name: default
Service Account: brupop-agent-service-account
Node: ip-10-10-110-247.ap-northeast-1.compute.internal/10.10.110.247
Start Time: Mon, 19 Sep 2022 00:59:01 +0900
Labels: brupop.bottlerocket.aws/component=agent
controller-revision-hash=6fc8c5cbb5
pod-template-generation=4
Annotations: kubernetes.io/psp: eks.privileged
Status: Running
IP: 10.10.101.170
IPs:
IP: 10.10.101.170
Controlled By: DaemonSet/brupop-agent
Containers:
brupop:
Container ID: containerd://093d067019f4cf4bed00d4c3d87aa8516f25afcabfc3f867281df4e68bf1dbe1
Image: public.ecr.aws/bottlerocket/bottlerocket-update-operator:v0.2.2
Image ID: public.ecr.aws/bottlerocket/bottlerocket-update-operator@sha256:a6de31e1b3553e0c5b5401ec7f7cc435c150481f5c4827c061e523106b9748c0
Port: <none>
Host Port: <none>
Command:
./agent
State: Running
Started: Mon, 19 Sep 2022 00:59:10 +0900
Ready: True
Restart Count: 0
Limits:
memory: 50Mi
Requests:
cpu: 10m
memory: 50Mi
Environment:
RUST_LOG: info,agent::apiclient=warn,agent::agentclient=warn
MY_NODE_NAME: (v1:spec.nodeName)
Mounts:
/bin/apiclient from bottlerocket-apiclient (rw)
/etc/brupop-tls-keys from bottlerocket-tls-keys (rw)
/run/api.sock from bottlerocket-api-socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-555bw (ro)
/var/run/secrets/tokens/ from bottlerocket-agent-service-account-token (rw)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
bottlerocket-api-socket:
Type: HostPath (bare host directory volume)
Path: /run/api.sock
HostPathType: Socket
bottlerocket-apiclient:
Type: HostPath (bare host directory volume)
Path: /bin/apiclient
HostPathType: File
bottlerocket-agent-service-account-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3600
bottlerocket-tls-keys:
Type: Secret (a volume populated by a Secret)
SecretName: brupop-tls
Optional: false
kube-api-access-555bw:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events: <none>
Name: brupop-agent-fvbpn
Namespace: brupop-bottlerocket-aws
Priority: 500000
Priority Class Name: default
Service Account: brupop-agent-service-account
Node: ip-10-10-131-243.ap-northeast-1.compute.internal/10.10.131.243
Start Time: Thu, 15 Sep 2022 04:28:05 +0900
Labels: brupop.bottlerocket.aws/component=agent
controller-revision-hash=6fc8c5cbb5
pod-template-generation=4
Annotations: kubernetes.io/psp: eks.privileged
Status: Running
IP: 10.10.157.221
IPs:
IP: 10.10.157.221
Controlled By: DaemonSet/brupop-agent
Containers:
brupop:
Container ID: containerd://9de6674ac3bb5b4133d7c13c262a7d1d413d6e730eace52fa4d38f5bf41e22f3
Image: public.ecr.aws/bottlerocket/bottlerocket-update-operator:v0.2.2
Image ID: public.ecr.aws/bottlerocket/bottlerocket-update-operator@sha256:a6de31e1b3553e0c5b5401ec7f7cc435c150481f5c4827c061e523106b9748c0
Port: <none>
Host Port: <none>
Command:
./agent
State: Running
Started: Thu, 15 Sep 2022 04:28:12 +0900
Ready: True
Restart Count: 0
Limits:
memory: 50Mi
Requests:
cpu: 10m
memory: 50Mi
Environment:
RUST_LOG: info,agent::apiclient=warn,agent::agentclient=warn
MY_NODE_NAME: (v1:spec.nodeName)
Mounts:
/bin/apiclient from bottlerocket-apiclient (rw)
/etc/brupop-tls-keys from bottlerocket-tls-keys (rw)
/run/api.sock from bottlerocket-api-socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-96gvl (ro)
/var/run/secrets/tokens/ from bottlerocket-agent-service-account-token (rw)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
bottlerocket-api-socket:
Type: HostPath (bare host directory volume)
Path: /run/api.sock
HostPathType: Socket
bottlerocket-apiclient:
Type: HostPath (bare host directory volume)
Path: /bin/apiclient
HostPathType: File
bottlerocket-agent-service-account-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3600
bottlerocket-tls-keys:
Type: Secret (a volume populated by a Secret)
SecretName: brupop-tls
Optional: false
kube-api-access-96gvl:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events: <none>
Name: brupop-agent-km7jl
Namespace: brupop-bottlerocket-aws
Priority: 500000
Priority Class Name: default
Service Account: brupop-agent-service-account
Node: ip-10-10-99-185.ap-northeast-1.compute.internal/10.10.99.185
Start Time: Wed, 21 Sep 2022 10:41:21 +0900
Labels: brupop.bottlerocket.aws/component=agent
controller-revision-hash=6fc8c5cbb5
pod-template-generation=4
Annotations: kubernetes.io/psp: eks.privileged
Status: Running
IP: 10.10.101.107
IPs:
IP: 10.10.101.107
Controlled By: DaemonSet/brupop-agent
Containers:
brupop:
Container ID: containerd://16ca7bfafc9c0bc6bc56db4d172f42ebcde4fbacf7c56286896c93a50ba56669
Image: public.ecr.aws/bottlerocket/bottlerocket-update-operator:v0.2.2
Image ID: public.ecr.aws/bottlerocket/bottlerocket-update-operator@sha256:a6de31e1b3553e0c5b5401ec7f7cc435c150481f5c4827c061e523106b9748c0
Port: <none>
Host Port: <none>
Command:
./agent
State: Running
Started: Wed, 21 Sep 2022 10:41:27 +0900
Ready: True
Restart Count: 0
Limits:
memory: 50Mi
Requests:
cpu: 10m
memory: 50Mi
Environment:
RUST_LOG: info,agent::apiclient=warn,agent::agentclient=warn
MY_NODE_NAME: (v1:spec.nodeName)
Mounts:
/bin/apiclient from bottlerocket-apiclient (rw)
/etc/brupop-tls-keys from bottlerocket-tls-keys (rw)
/run/api.sock from bottlerocket-api-socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-svpbs (ro)
/var/run/secrets/tokens/ from bottlerocket-agent-service-account-token (rw)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
bottlerocket-api-socket:
Type: HostPath (bare host directory volume)
Path: /run/api.sock
HostPathType: Socket
bottlerocket-apiclient:
Type: HostPath (bare host directory volume)
Path: /bin/apiclient
HostPathType: File
bottlerocket-agent-service-account-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3600
bottlerocket-tls-keys:
Type: Secret (a volume populated by a Secret)
SecretName: brupop-tls
Optional: false
kube-api-access-svpbs:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events: <none>
Name: brupop-agent-msqxt
Namespace: brupop-bottlerocket-aws
Priority: 500000
Priority Class Name: default
Service Account: brupop-agent-service-account
Node: ip-10-10-125-43.ap-northeast-1.compute.internal/10.10.125.43
Start Time: Wed, 21 Sep 2022 09:40:22 +0900
Labels: brupop.bottlerocket.aws/component=agent
controller-revision-hash=6fc8c5cbb5
pod-template-generation=4
Annotations: kubernetes.io/psp: eks.privileged
Status: Running
IP: 10.10.106.98
IPs:
IP: 10.10.106.98
Controlled By: DaemonSet/brupop-agent
Containers:
brupop:
Container ID: containerd://184ce2d2a6d58f5989fca3b86434d01214936df4e2383d6f98b5557851f04f78
Image: public.ecr.aws/bottlerocket/bottlerocket-update-operator:v0.2.2
Image ID: public.ecr.aws/bottlerocket/bottlerocket-update-operator@sha256:a6de31e1b3553e0c5b5401ec7f7cc435c150481f5c4827c061e523106b9748c0
Port: <none>
Host Port: <none>
Command:
./agent
State: Running
Started: Wed, 21 Sep 2022 09:40:27 +0900
Ready: True
Restart Count: 0
Limits:
memory: 50Mi
Requests:
cpu: 10m
memory: 50Mi
Environment:
RUST_LOG: info,agent::apiclient=warn,agent::agentclient=warn
MY_NODE_NAME: (v1:spec.nodeName)
Mounts:
/bin/apiclient from bottlerocket-apiclient (rw)
/etc/brupop-tls-keys from bottlerocket-tls-keys (rw)
/run/api.sock from bottlerocket-api-socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8mmfg (ro)
/var/run/secrets/tokens/ from bottlerocket-agent-service-account-token (rw)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
bottlerocket-api-socket:
Type: HostPath (bare host directory volume)
Path: /run/api.sock
HostPathType: Socket
bottlerocket-apiclient:
Type: HostPath (bare host directory volume)
Path: /bin/apiclient
HostPathType: File
bottlerocket-agent-service-account-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3600
bottlerocket-tls-keys:
Type: Secret (a volume populated by a Secret)
SecretName: brupop-tls
Optional: false
kube-api-access-8mmfg:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events: <none>
Name: brupop-agent-rfk6d
Namespace: brupop-bottlerocket-aws
Priority: 500000
Priority Class Name: default
Service Account: brupop-agent-service-account
Node: ip-10-10-60-250.ap-northeast-1.compute.internal/10.10.60.250
Start Time: Wed, 21 Sep 2022 09:34:57 +0900
Labels: brupop.bottlerocket.aws/component=agent
controller-revision-hash=6fc8c5cbb5
pod-template-generation=4
Annotations: kubernetes.io/psp: eks.privileged
Status: Running
IP: 10.10.34.74
IPs:
IP: 10.10.34.74
Controlled By: DaemonSet/brupop-agent
Containers:
brupop:
Container ID: containerd://55c6289981f38e082f6dbadb9abd619f8e60a8337629a5b071398af8591bc0f8
Image: public.ecr.aws/bottlerocket/bottlerocket-update-operator:v0.2.2
Image ID: public.ecr.aws/bottlerocket/bottlerocket-update-operator@sha256:a6de31e1b3553e0c5b5401ec7f7cc435c150481f5c4827c061e523106b9748c0
Port: <none>
Host Port: <none>
Command:
./agent
State: Running
Started: Wed, 21 Sep 2022 09:35:04 +0900
Ready: True
Restart Count: 0
Limits:
memory: 50Mi
Requests:
cpu: 10m
memory: 50Mi
Environment:
RUST_LOG: info,agent::apiclient=warn,agent::agentclient=warn
MY_NODE_NAME: (v1:spec.nodeName)
Mounts:
/bin/apiclient from bottlerocket-apiclient (rw)
/etc/brupop-tls-keys from bottlerocket-tls-keys (rw)
/run/api.sock from bottlerocket-api-socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tvs8j (ro)
/var/run/secrets/tokens/ from bottlerocket-agent-service-account-token (rw)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
bottlerocket-api-socket:
Type: HostPath (bare host directory volume)
Path: /run/api.sock
HostPathType: Socket
bottlerocket-apiclient:
Type: HostPath (bare host directory volume)
Path: /bin/apiclient
HostPathType: File
bottlerocket-agent-service-account-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3600
bottlerocket-tls-keys:
Type: Secret (a volume populated by a Secret)
SecretName: brupop-tls
Optional: false
kube-api-access-tvs8j:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events: <none>
Name: brupop-agent-t8nzq
Namespace: brupop-bottlerocket-aws
Priority: 500000
Priority Class Name: default
Service Account: brupop-agent-service-account
Node: ip-10-10-140-10.ap-northeast-1.compute.internal/10.10.140.10
Start Time: Sun, 11 Sep 2022 18:21:44 +0900
Labels: brupop.bottlerocket.aws/component=agent
controller-revision-hash=6fc8c5cbb5
pod-template-generation=4
Annotations: kubernetes.io/psp: eks.privileged
Status: Running
IP: 10.10.152.70
IPs:
IP: 10.10.152.70
Controlled By: DaemonSet/brupop-agent
Containers:
brupop:
Container ID: containerd://a8dc29018eccbca635da5b630ae529e7a3f67e0ea80d8ffbeacea0ad48e5dd11
Image: public.ecr.aws/bottlerocket/bottlerocket-update-operator:v0.2.2
Image ID: public.ecr.aws/bottlerocket/bottlerocket-update-operator@sha256:a6de31e1b3553e0c5b5401ec7f7cc435c150481f5c4827c061e523106b9748c0
Port: <none>
Host Port: <none>
Command:
./agent
State: Running
Started: Sun, 11 Sep 2022 18:21:51 +0900
Ready: True
Restart Count: 0
Limits:
memory: 50Mi
Requests:
cpu: 10m
memory: 50Mi
Environment:
RUST_LOG: info,agent::apiclient=warn,agent::agentclient=warn
MY_NODE_NAME: (v1:spec.nodeName)
Mounts:
/bin/apiclient from bottlerocket-apiclient (rw)
/etc/brupop-tls-keys from bottlerocket-tls-keys (rw)
/run/api.sock from bottlerocket-api-socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-msn4v (ro)
/var/run/secrets/tokens/ from bottlerocket-agent-service-account-token (rw)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
bottlerocket-api-socket:
Type: HostPath (bare host directory volume)
Path: /run/api.sock
HostPathType: Socket
bottlerocket-apiclient:
Type: HostPath (bare host directory volume)
Path: /bin/apiclient
HostPathType: File
bottlerocket-agent-service-account-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3600
bottlerocket-tls-keys:
Type: Secret (a volume populated by a Secret)
SecretName: brupop-tls
Optional: false
kube-api-access-msn4v:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events: <none>
Name: brupop-agent-vrfgd
Namespace: brupop-bottlerocket-aws
Priority: 500000
Priority Class Name: default
Service Account: brupop-agent-service-account
Node: ip-10-10-150-7.ap-northeast-1.compute.internal/10.10.150.7
Start Time: Wed, 21 Sep 2022 10:30:36 +0900
Labels: brupop.bottlerocket.aws/component=agent
controller-revision-hash=6fc8c5cbb5
pod-template-generation=4
Annotations: kubernetes.io/psp: eks.privileged
Status: Running
IP: 10.10.155.235
IPs:
IP: 10.10.155.235
Controlled By: DaemonSet/brupop-agent
Containers:
brupop:
Container ID: containerd://395958ec68cece5a7308882119f9cee8c35739d9040ab2d507632ef55ac71a4b
Image: public.ecr.aws/bottlerocket/bottlerocket-update-operator:v0.2.2
Image ID: public.ecr.aws/bottlerocket/bottlerocket-update-operator@sha256:a6de31e1b3553e0c5b5401ec7f7cc435c150481f5c4827c061e523106b9748c0
Port: <none>
Host Port: <none>
Command:
./agent
State: Running
Started: Wed, 21 Sep 2022 10:30:42 +0900
Ready: True
Restart Count: 0
Limits:
memory: 50Mi
Requests:
cpu: 10m
memory: 50Mi
Environment:
RUST_LOG: info,agent::apiclient=warn,agent::agentclient=warn
MY_NODE_NAME: (v1:spec.nodeName)
Mounts:
/bin/apiclient from bottlerocket-apiclient (rw)
/etc/brupop-tls-keys from bottlerocket-tls-keys (rw)
/run/api.sock from bottlerocket-api-socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lqwzz (ro)
/var/run/secrets/tokens/ from bottlerocket-agent-service-account-token (rw)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
bottlerocket-api-socket:
Type: HostPath (bare host directory volume)
Path: /run/api.sock
HostPathType: Socket
bottlerocket-apiclient:
Type: HostPath (bare host directory volume)
Path: /bin/apiclient
HostPathType: File
bottlerocket-agent-service-account-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3600
bottlerocket-tls-keys:
Type: Secret (a volume populated by a Secret)
SecretName: brupop-tls
Optional: false
kube-api-access-lqwzz:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events: <none>
Name: brupop-agent-xzpsk
Namespace: brupop-bottlerocket-aws
Priority: 500000
Priority Class Name: default
Service Account: brupop-agent-service-account
Node: ip-10-10-50-25.ap-northeast-1.compute.internal/10.10.50.25
Start Time: Wed, 21 Sep 2022 10:46:49 +0900
Labels: brupop.bottlerocket.aws/component=agent
controller-revision-hash=6fc8c5cbb5
pod-template-generation=4
Annotations: kubernetes.io/psp: eks.privileged
Status: Running
IP: 10.10.49.212
IPs:
IP: 10.10.49.212
Controlled By: DaemonSet/brupop-agent
Containers:
brupop:
Container ID: containerd://fa204999e73a70ed4ce5c24aa51b79e442e46dcec5188b06ed7863ff8eceb194
Image: public.ecr.aws/bottlerocket/bottlerocket-update-operator:v0.2.2
Image ID: public.ecr.aws/bottlerocket/bottlerocket-update-operator@sha256:a6de31e1b3553e0c5b5401ec7f7cc435c150481f5c4827c061e523106b9748c0
Port: <none>
Host Port: <none>
Command:
./agent
State: Running
Started: Wed, 21 Sep 2022 10:46:55 +0900
Ready: True
Restart Count: 0
Limits:
memory: 50Mi
Requests:
cpu: 10m
memory: 50Mi
Environment:
RUST_LOG: info,agent::apiclient=warn,agent::agentclient=warn
MY_NODE_NAME: (v1:spec.nodeName)
Mounts:
/bin/apiclient from bottlerocket-apiclient (rw)
/etc/brupop-tls-keys from bottlerocket-tls-keys (rw)
/run/api.sock from bottlerocket-api-socket (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2x6ff (ro)
/var/run/secrets/tokens/ from bottlerocket-agent-service-account-token (rw)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
bottlerocket-api-socket:
Type: HostPath (bare host directory volume)
Path: /run/api.sock
HostPathType: Socket
bottlerocket-apiclient:
Type: HostPath (bare host directory volume)
Path: /bin/apiclient
HostPathType: File
bottlerocket-agent-service-account-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3600
bottlerocket-tls-keys:
Type: Secret (a volume populated by a Secret)
SecretName: brupop-tls
Optional: false
kube-api-access-2x6ff:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events: <none>
Name: brupop-apiserver-8575cc4658-4mm6p
Namespace: brupop-bottlerocket-aws
Priority: 500000
Priority Class Name: default
Service Account: brupop-apiserver-service-account
Node: ip-10-10-50-25.ap-northeast-1.compute.internal/10.10.50.25
Start Time: Wed, 21 Sep 2022 11:18:14 +0900
Labels: brupop.bottlerocket.aws/component=apiserver
pod-template-hash=8575cc4658
Annotations: kubectl.kubernetes.io/restartedAt: 2022-09-21T11:18:13+09:00
kubernetes.io/psp: eks.privileged
Status: Running
IP: 10.10.35.231
IPs:
IP: 10.10.35.231
Controlled By: ReplicaSet/brupop-apiserver-8575cc4658
Containers:
brupop:
Container ID: containerd://1945dfc4d8a135be9d44a3cfcafe2702fcacdebaed1adf5c58fda3a6a856cd9e
Image: public.ecr.aws/bottlerocket/bottlerocket-update-operator:v0.2.2
Image ID: public.ecr.aws/bottlerocket/bottlerocket-update-operator@sha256:a6de31e1b3553e0c5b5401ec7f7cc435c150481f5c4827c061e523106b9748c0
Port: 8443/TCP
Host Port: 0/TCP
Command:
./apiserver
State: Running
Started: Wed, 21 Sep 2022 11:18:14 +0900
Ready: True
Restart Count: 0
Liveness: http-get https://:8443/ping delay=5s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get https://:8443/ping delay=5s timeout=1s period=10s #success=1 #failure=3
Environment:
RUST_LOG: info
Mounts:
/etc/brupop-tls-keys from bottlerocket-tls-keys (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-x74zk (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
bottlerocket-tls-keys:
Type: Secret (a volume populated by a Secret)
SecretName: brupop-tls
Optional: false
kube-api-access-x74zk:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events: <none>
Name: brupop-apiserver-8575cc4658-jjkkv
Namespace: brupop-bottlerocket-aws
Priority: 500000
Priority Class Name: default
Service Account: brupop-apiserver-service-account
Node: ip-10-10-99-185.ap-northeast-1.compute.internal/10.10.99.185
Start Time: Wed, 21 Sep 2022 11:18:34 +0900
Labels: brupop.bottlerocket.aws/component=apiserver
pod-template-hash=8575cc4658
Annotations: kubectl.kubernetes.io/restartedAt: 2022-09-21T11:18:13+09:00
kubernetes.io/psp: eks.privileged
Status: Running
IP: 10.10.101.84
IPs:
IP: 10.10.101.84
Controlled By: ReplicaSet/brupop-apiserver-8575cc4658
Containers:
brupop:
Container ID: containerd://6740ae2b280e9835e33856fad20f88e22393f609167b3c15547f92d4e78e5479
Image: public.ecr.aws/bottlerocket/bottlerocket-update-operator:v0.2.2
Image ID: public.ecr.aws/bottlerocket/bottlerocket-update-operator@sha256:a6de31e1b3553e0c5b5401ec7f7cc435c150481f5c4827c061e523106b9748c0
Port: 8443/TCP
Host Port: 0/TCP
Command:
./apiserver
State: Running
Started: Wed, 21 Sep 2022 11:18:35 +0900
Ready: True
Restart Count: 0
Liveness: http-get https://:8443/ping delay=5s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get https://:8443/ping delay=5s timeout=1s period=10s #success=1 #failure=3
Environment:
RUST_LOG: info
Mounts:
/etc/brupop-tls-keys from bottlerocket-tls-keys (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-k9cwf (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
bottlerocket-tls-keys:
Type: Secret (a volume populated by a Secret)
SecretName: brupop-tls
Optional: false
kube-api-access-k9cwf:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events: <none>
Name: brupop-apiserver-8575cc4658-w77sx
Namespace: brupop-bottlerocket-aws
Priority: 500000
Priority Class Name: default
Service Account: brupop-apiserver-service-account
Node: ip-10-10-152-12.ap-northeast-1.compute.internal/10.10.152.12
Start Time: Wed, 21 Sep 2022 11:18:24 +0900
Labels: brupop.bottlerocket.aws/component=apiserver
pod-template-hash=8575cc4658
Annotations: kubectl.kubernetes.io/restartedAt: 2022-09-21T11:18:13+09:00
kubernetes.io/psp: eks.privileged
Status: Running
IP: 10.10.130.71
IPs:
IP: 10.10.130.71
Controlled By: ReplicaSet/brupop-apiserver-8575cc4658
Containers:
brupop:
Container ID: containerd://bbb13eda91563ae8713a5102349e1a8804af8ca6e5a912c0b22e26d51ea7a529
Image: public.ecr.aws/bottlerocket/bottlerocket-update-operator:v0.2.2
Image ID: public.ecr.aws/bottlerocket/bottlerocket-update-operator@sha256:a6de31e1b3553e0c5b5401ec7f7cc435c150481f5c4827c061e523106b9748c0
Port: 8443/TCP
Host Port: 0/TCP
Command:
./apiserver
State: Running
Started: Wed, 21 Sep 2022 11:18:24 +0900
Ready: True
Restart Count: 0
Liveness: http-get https://:8443/ping delay=5s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get https://:8443/ping delay=5s timeout=1s period=10s #success=1 #failure=3
Environment:
RUST_LOG: info
Mounts:
/etc/brupop-tls-keys from bottlerocket-tls-keys (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6ctzb (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
bottlerocket-tls-keys:
Type: Secret (a volume populated by a Secret)
SecretName: brupop-tls
Optional: false
kube-api-access-6ctzb:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events: <none>
Name: brupop-controller-deployment-74d457b69d-6sb2t
Namespace: brupop-bottlerocket-aws
Priority: 1000000
Priority Class Name: brupop-controller-high-priority
Service Account: brupop-controller-service-account
Node: ip-10-10-150-7.ap-northeast-1.compute.internal/10.10.150.7
Start Time: Wed, 21 Sep 2022 10:34:56 +0900
Labels: brupop.bottlerocket.aws/component=brupop-controller
pod-template-hash=74d457b69d
Annotations: kubernetes.io/psp: eks.privileged
Status: Running
IP: 10.10.156.79
IPs:
IP: 10.10.156.79
Controlled By: ReplicaSet/brupop-controller-deployment-74d457b69d
Containers:
brupop:
Container ID: containerd://2dec4c8ed3e36606f5221b226a5172ba00e7a81488cb0b7c332f9bb85675312a
Image: public.ecr.aws/bottlerocket/bottlerocket-update-operator:v0.2.2
Image ID: public.ecr.aws/bottlerocket/bottlerocket-update-operator@sha256:a6de31e1b3553e0c5b5401ec7f7cc435c150481f5c4827c061e523106b9748c0
Port: <none>
Host Port: <none>
Command:
./controller
State: Running
Started: Wed, 21 Sep 2022 10:34:58 +0900
Ready: True
Restart Count: 0
Environment:
RUST_LOG: info,controller::controller=warn,controller::statemachine=warn
MY_NODE_NAME: (v1:spec.nodeName)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fhnwr (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-fhnwr:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events: <none>
All the pods are using the v0.2.2 image.
$ kubectl describe pods -n brupop-bottlerocket-aws | grep Image: | uniq
Image: public.ecr.aws/bottlerocket/bottlerocket-update-operator:v0.2.2
Could you please check the logs from the agent and controller to see if any of those requesting v1?
You can use kubectl logs $pod_name to check the log.
I wonder if a force pull and re-deployment could mitigate the issue by adding imagePullPolicy: Always to the corresponding deployment, for example:
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
app.kubernetes.io/component: agent
app.kubernetes.io/managed-by: brupop
app.kubernetes.io/part-of: brupop
brupop.bottlerocket.aws/component: agent
name: brupop-agent
namespace: brupop-bottlerocket-aws
spec:
selector:
matchLabels:
brupop.bottlerocket.aws/component: agent
template:
metadata:
labels:
brupop.bottlerocket.aws/component: agent
namespace: brupop-bottlerocket-aws
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
- key: bottlerocket.aws/updater-interface-version
operator: In
values:
- 2.0.0
- key: kubernetes.io/arch
operator: In
values:
- amd64
- arm64
containers:
- command:
- "./agent"
env:
- name: MY_NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: EXCLUDE_FROM_LB_WAIT_TIME_IN_SEC
value: "0"
image: "public.ecr.aws/bottlerocket/bottlerocket-update-operator:v0.2.2"
imagePullPolicy: Always <---------------force pull the image for agent
name: brupop
resources:
limits:
memory: 50Mi
requests:
cpu: 10m
memory: 50Mi
...
I've changed the policy to Always, restarted all the pods.
I didn't observe any change.
Here are the recent logs of the agent and controller.
{"v":0,"name":"agent","msg":"[SHADOW_STATUS_WITH_REFRESHED_SYSTEM_MATADATA - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:25.539625237+00:00","target":"agent::agentclient","line":179,"file":"agent/src/agentclient.rs","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","elapsed_milliseconds":20023}
{"v":0,"name":"agent","msg":"[UPDATE_STATUS_IN_SHADOW - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:25.539678111+00:00","target":"agent::agentclient","line":300,"file":"agent/src/agentclient.rs","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","elapsed_milliseconds":20024}
{"v":0,"name":"agent","msg":"[CREATE_SHADOW_IF_NOT_EXIST - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:30.541721079+00:00","target":"agent::agentclient","line":282,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[CHECK_NODE_SHADOW_EXISTS - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:30.541757330+00:00","target":"agent::agentclient","line":82,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[CHECK_NODE_SHADOW_EXISTS - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:30.541771242+00:00","target":"agent::agentclient","line":82,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[CREATE_SHADOW_IF_NOT_EXIST - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:30.541781957+00:00","target":"agent::agentclient","line":282,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[INITIALIZE_SHADOW_IF_NOT_INITIALIZED - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:30.541789634+00:00","target":"agent::agentclient","line":291,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[CHECK_SHADOW_STATUS_EXISTS - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:30.541796551+00:00","target":"agent::agentclient","line":125,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[FETCH_SHADOW - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:30.541804728+00:00","target":"agent::agentclient","line":157,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[FETCH_SHADOW - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:30.541827127+00:00","target":"agent::agentclient","line":157,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[CHECK_SHADOW_STATUS_EXISTS - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:30.541838720+00:00","target":"agent::agentclient","line":125,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[INITIALIZE_SHADOW_IF_NOT_INITIALIZED - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:30.541848994+00:00","target":"agent::agentclient","line":291,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[FETCH_SHADOW - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:30.541855331+00:00","target":"agent::agentclient","line":157,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[FETCH_SHADOW - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:30.541868460+00:00","target":"agent::agentclient","line":157,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[HANDLE_STATE_TRANSITION - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:30.541941548+00:00","target":"agent::agentclient","line":331,"file":"agent/src/agentclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }"}
{"v":0,"name":"agent","msg":"[HANDLE_STATE_TRANSITION - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:30.541962779+00:00","target":"agent::agentclient","line":331,"file":"agent/src/agentclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[UPDATE_STATUS_IN_SHADOW - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:30.542027477+00:00","target":"agent::agentclient","line":300,"file":"agent/src/agentclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }"}
{"v":0,"name":"agent","msg":"[SHADOW_STATUS_WITH_REFRESHED_SYSTEM_MATADATA - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:30.542049498+00:00","target":"agent::agentclient","line":179,"file":"agent/src/agentclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }"}
{"v":0,"name":"agent","msg":"[SHADOW_STATUS_WITH_REFRESHED_SYSTEM_MATADATA - EVENT] API server busy, retrying later ...","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:30.553491697+00:00","target":"agent::apiclient","line":140,"file":"agent/src/apiclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }"}
{"v":0,"name":"agent","msg":"[SHADOW_STATUS_WITH_REFRESHED_SYSTEM_MATADATA - EVENT] API server busy, retrying later ...","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:40.560912897+00:00","target":"agent::apiclient","line":140,"file":"agent/src/apiclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }"}
{"v":0,"name":"agent","msg":"[SHADOW_STATUS_WITH_REFRESHED_SYSTEM_MATADATA - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:50.565715478+00:00","target":"agent::agentclient","line":179,"file":"agent/src/agentclient.rs","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","elapsed_milliseconds":20023}
{"v":0,"name":"agent","msg":"[UPDATE_STATUS_IN_SHADOW - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:50.565771823+00:00","target":"agent::agentclient","line":300,"file":"agent/src/agentclient.rs","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","elapsed_milliseconds":20023}
{"v":0,"name":"agent","msg":"[CREATE_SHADOW_IF_NOT_EXIST - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:55.567512297+00:00","target":"agent::agentclient","line":282,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[CHECK_NODE_SHADOW_EXISTS - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:55.567563868+00:00","target":"agent::agentclient","line":82,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[CHECK_NODE_SHADOW_EXISTS - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:55.567608798+00:00","target":"agent::agentclient","line":82,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[CREATE_SHADOW_IF_NOT_EXIST - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:55.567626286+00:00","target":"agent::agentclient","line":282,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[INITIALIZE_SHADOW_IF_NOT_INITIALIZED - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:55.567637227+00:00","target":"agent::agentclient","line":291,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[CHECK_SHADOW_STATUS_EXISTS - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:55.567647972+00:00","target":"agent::agentclient","line":125,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[FETCH_SHADOW - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:55.567661497+00:00","target":"agent::agentclient","line":157,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[FETCH_SHADOW - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:55.567686759+00:00","target":"agent::agentclient","line":157,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[CHECK_SHADOW_STATUS_EXISTS - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:55.567702441+00:00","target":"agent::agentclient","line":125,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[INITIALIZE_SHADOW_IF_NOT_INITIALIZED - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:55.567712589+00:00","target":"agent::agentclient","line":291,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[FETCH_SHADOW - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:55.567724012+00:00","target":"agent::agentclient","line":157,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[FETCH_SHADOW - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:55.567743393+00:00","target":"agent::agentclient","line":157,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[HANDLE_STATE_TRANSITION - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:55.567842923+00:00","target":"agent::agentclient","line":331,"file":"agent/src/agentclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }"}
{"v":0,"name":"agent","msg":"[HANDLE_STATE_TRANSITION - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:55.567868087+00:00","target":"agent::agentclient","line":331,"file":"agent/src/agentclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[UPDATE_STATUS_IN_SHADOW - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:55.567943143+00:00","target":"agent::agentclient","line":300,"file":"agent/src/agentclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }"}
{"v":0,"name":"agent","msg":"[SHADOW_STATUS_WITH_REFRESHED_SYSTEM_MATADATA - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:55.567969132+00:00","target":"agent::agentclient","line":179,"file":"agent/src/agentclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }"}
{"v":0,"name":"agent","msg":"[SHADOW_STATUS_WITH_REFRESHED_SYSTEM_MATADATA - EVENT] API server busy, retrying later ...","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:06:55.579225253+00:00","target":"agent::apiclient","line":140,"file":"agent/src/apiclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }"}
{"v":0,"name":"agent","msg":"[SHADOW_STATUS_WITH_REFRESHED_SYSTEM_MATADATA - EVENT] API server busy, retrying later ...","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:05.586638781+00:00","target":"agent::apiclient","line":140,"file":"agent/src/apiclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }"}
{"v":0,"name":"agent","msg":"[SHADOW_STATUS_WITH_REFRESHED_SYSTEM_MATADATA - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:15.591461398+00:00","target":"agent::agentclient","line":179,"file":"agent/src/agentclient.rs","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","elapsed_milliseconds":20023}
{"v":0,"name":"agent","msg":"[UPDATE_STATUS_IN_SHADOW - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:15.591523039+00:00","target":"agent::agentclient","line":300,"file":"agent/src/agentclient.rs","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","elapsed_milliseconds":20023}
{"v":0,"name":"agent","msg":"[CREATE_SHADOW_IF_NOT_EXIST - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:20.592721336+00:00","target":"agent::agentclient","line":282,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[CHECK_NODE_SHADOW_EXISTS - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:20.592762879+00:00","target":"agent::agentclient","line":82,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[CHECK_NODE_SHADOW_EXISTS - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:20.592776413+00:00","target":"agent::agentclient","line":82,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[CREATE_SHADOW_IF_NOT_EXIST - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:20.592786880+00:00","target":"agent::agentclient","line":282,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[INITIALIZE_SHADOW_IF_NOT_INITIALIZED - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:20.592795544+00:00","target":"agent::agentclient","line":291,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[CHECK_SHADOW_STATUS_EXISTS - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:20.592804978+00:00","target":"agent::agentclient","line":125,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[FETCH_SHADOW - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:20.592815905+00:00","target":"agent::agentclient","line":157,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[FETCH_SHADOW - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:20.592842180+00:00","target":"agent::agentclient","line":157,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[CHECK_SHADOW_STATUS_EXISTS - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:20.592857515+00:00","target":"agent::agentclient","line":125,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[INITIALIZE_SHADOW_IF_NOT_INITIALIZED - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:20.592869323+00:00","target":"agent::agentclient","line":291,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[FETCH_SHADOW - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:20.592882713+00:00","target":"agent::agentclient","line":157,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[FETCH_SHADOW - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:20.592900931+00:00","target":"agent::agentclient","line":157,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[HANDLE_STATE_TRANSITION - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:20.592978788+00:00","target":"agent::agentclient","line":331,"file":"agent/src/agentclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }"}
{"v":0,"name":"agent","msg":"[HANDLE_STATE_TRANSITION - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:20.593002581+00:00","target":"agent::agentclient","line":331,"file":"agent/src/agentclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[UPDATE_STATUS_IN_SHADOW - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:20.593079815+00:00","target":"agent::agentclient","line":300,"file":"agent/src/agentclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }"}
{"v":0,"name":"agent","msg":"[SHADOW_STATUS_WITH_REFRESHED_SYSTEM_MATADATA - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:20.593104949+00:00","target":"agent::agentclient","line":179,"file":"agent/src/agentclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }"}
{"v":0,"name":"agent","msg":"[SHADOW_STATUS_WITH_REFRESHED_SYSTEM_MATADATA - EVENT] API server busy, retrying later ...","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:20.604311056+00:00","target":"agent::apiclient","line":140,"file":"agent/src/apiclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }"}
{"v":0,"name":"agent","msg":"[SHADOW_STATUS_WITH_REFRESHED_SYSTEM_MATADATA - EVENT] API server busy, retrying later ...","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:30.611832902+00:00","target":"agent::apiclient","line":140,"file":"agent/src/apiclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }"}
{"v":0,"name":"agent","msg":"[SHADOW_STATUS_WITH_REFRESHED_SYSTEM_MATADATA - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:40.615950037+00:00","target":"agent::agentclient","line":179,"file":"agent/src/agentclient.rs","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","elapsed_milliseconds":20022}
{"v":0,"name":"agent","msg":"[UPDATE_STATUS_IN_SHADOW - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:40.616004075+00:00","target":"agent::agentclient","line":300,"file":"agent/src/agentclient.rs","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","elapsed_milliseconds":20022}
{"v":0,"name":"agent","msg":"[CREATE_SHADOW_IF_NOT_EXIST - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:45.617068142+00:00","target":"agent::agentclient","line":282,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[CHECK_NODE_SHADOW_EXISTS - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:45.617106683+00:00","target":"agent::agentclient","line":82,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[CHECK_NODE_SHADOW_EXISTS - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:45.617123676+00:00","target":"agent::agentclient","line":82,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[CREATE_SHADOW_IF_NOT_EXIST - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:45.617138080+00:00","target":"agent::agentclient","line":282,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[INITIALIZE_SHADOW_IF_NOT_INITIALIZED - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:45.617151457+00:00","target":"agent::agentclient","line":291,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[CHECK_SHADOW_STATUS_EXISTS - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:45.617162705+00:00","target":"agent::agentclient","line":125,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[FETCH_SHADOW - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:45.617178390+00:00","target":"agent::agentclient","line":157,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[FETCH_SHADOW - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:45.617205521+00:00","target":"agent::agentclient","line":157,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[CHECK_SHADOW_STATUS_EXISTS - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:45.617220855+00:00","target":"agent::agentclient","line":125,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[INITIALIZE_SHADOW_IF_NOT_INITIALIZED - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:45.617236232+00:00","target":"agent::agentclient","line":291,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[FETCH_SHADOW - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:45.617246593+00:00","target":"agent::agentclient","line":157,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[FETCH_SHADOW - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:45.617266527+00:00","target":"agent::agentclient","line":157,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[HANDLE_STATE_TRANSITION - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:45.617346386+00:00","target":"agent::agentclient","line":331,"file":"agent/src/agentclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }"}
{"v":0,"name":"agent","msg":"[HANDLE_STATE_TRANSITION - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:45.617392276+00:00","target":"agent::agentclient","line":331,"file":"agent/src/agentclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[UPDATE_STATUS_IN_SHADOW - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:45.617471872+00:00","target":"agent::agentclient","line":300,"file":"agent/src/agentclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }"}
{"v":0,"name":"agent","msg":"[SHADOW_STATUS_WITH_REFRESHED_SYSTEM_MATADATA - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:45.617498487+00:00","target":"agent::agentclient","line":179,"file":"agent/src/agentclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }"}
{"v":0,"name":"agent","msg":"[SHADOW_STATUS_WITH_REFRESHED_SYSTEM_MATADATA - EVENT] API server busy, retrying later ...","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:45.629319895+00:00","target":"agent::apiclient","line":140,"file":"agent/src/apiclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }"}
{"v":0,"name":"agent","msg":"[SHADOW_STATUS_WITH_REFRESHED_SYSTEM_MATADATA - EVENT] API server busy, retrying later ...","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:07:55.637205603+00:00","target":"agent::apiclient","line":140,"file":"agent/src/apiclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }"}
{"v":0,"name":"agent","msg":"[SHADOW_STATUS_WITH_REFRESHED_SYSTEM_MATADATA - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:08:05.641432857+00:00","target":"agent::agentclient","line":179,"file":"agent/src/agentclient.rs","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","elapsed_milliseconds":20023}
{"v":0,"name":"agent","msg":"[UPDATE_STATUS_IN_SHADOW - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:08:05.641489401+00:00","target":"agent::agentclient","line":300,"file":"agent/src/agentclient.rs","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","elapsed_milliseconds":20023}
{"v":0,"name":"agent","msg":"[CREATE_SHADOW_IF_NOT_EXIST - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:08:10.642539061+00:00","target":"agent::agentclient","line":282,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[CHECK_NODE_SHADOW_EXISTS - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:08:10.642578051+00:00","target":"agent::agentclient","line":82,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[CHECK_NODE_SHADOW_EXISTS - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:08:10.642591861+00:00","target":"agent::agentclient","line":82,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[CREATE_SHADOW_IF_NOT_EXIST - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:08:10.642601516+00:00","target":"agent::agentclient","line":282,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[INITIALIZE_SHADOW_IF_NOT_INITIALIZED - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:08:10.642608575+00:00","target":"agent::agentclient","line":291,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[CHECK_SHADOW_STATUS_EXISTS - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:08:10.642616585+00:00","target":"agent::agentclient","line":125,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[FETCH_SHADOW - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:08:10.642626415+00:00","target":"agent::agentclient","line":157,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[FETCH_SHADOW - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:08:10.642651731+00:00","target":"agent::agentclient","line":157,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[CHECK_SHADOW_STATUS_EXISTS - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:08:10.642664545+00:00","target":"agent::agentclient","line":125,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[INITIALIZE_SHADOW_IF_NOT_INITIALIZED - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:08:10.642671869+00:00","target":"agent::agentclient","line":291,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[FETCH_SHADOW - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:08:10.642677823+00:00","target":"agent::agentclient","line":157,"file":"agent/src/agentclient.rs"}
{"v":0,"name":"agent","msg":"[FETCH_SHADOW - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:08:10.642689298+00:00","target":"agent::agentclient","line":157,"file":"agent/src/agentclient.rs","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[HANDLE_STATE_TRANSITION - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:08:10.642763457+00:00","target":"agent::agentclient","line":331,"file":"agent/src/agentclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }"}
{"v":0,"name":"agent","msg":"[HANDLE_STATE_TRANSITION - END]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:08:10.642792628+00:00","target":"agent::agentclient","line":331,"file":"agent/src/agentclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","elapsed_milliseconds":0}
{"v":0,"name":"agent","msg":"[UPDATE_STATUS_IN_SHADOW - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:08:10.642859484+00:00","target":"agent::agentclient","line":300,"file":"agent/src/agentclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }"}
{"v":0,"name":"agent","msg":"[SHADOW_STATUS_WITH_REFRESHED_SYSTEM_MATADATA - START]","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:08:10.642881984+00:00","target":"agent::agentclient","line":179,"file":"agent/src/agentclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }"}
{"v":0,"name":"agent","msg":"[SHADOW_STATUS_WITH_REFRESHED_SYSTEM_MATADATA - EVENT] API server busy, retrying later ...","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:08:10.655096617+00:00","target":"agent::apiclient","line":140,"file":"agent/src/apiclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }"}
{"v":0,"name":"agent","msg":"[SHADOW_STATUS_WITH_REFRESHED_SYSTEM_MATADATA - EVENT] API server busy, retrying later ...","level":30,"hostname":"brupop-agent-4dtm6","pid":1,"time":"2022-09-22T08:08:20.662945295+00:00","target":"agent::apiclient","line":140,"file":"agent/src/apiclient.rs","bottlerocket_shadow":"BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }","state":"Idle","shadow_error_info":"ShadowErrorInfo { crash_count: 0, state_transition_failure_timestamp: None }"}
{"v":0,"name":"brupop-controller","msg":"[GET_ASSOCIATED_BOTTLEROCKETSHADOW_NAME - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.604821292+00:00","target":"controller::controller","line":218,"file":"controller/src/controller.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[SORT_SHADOWS - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605145063+00:00","target":"controller::controller","line":236,"file":"controller/src/controller.rs","associated_brs_name":"brs-ip-10-10-131-243.ap-northeast-1.compute.internal","shadows":"[BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T01:30:42Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"b24116e7-9e56-41a0-9a4d-5df1650ff6fe\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:30:42Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:31:03Z)) }]), name: Some(\"brs-ip-10-10-150-7.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-150-7.ap-northeast-1.compute.internal\", uid: \"b24116e7-9e56-41a0-9a4d-5df1650ff6fe\" }]), resource_version: Some(\"758613376\"), self_link: None, uid: Some(\"c6c3b44b-25ee-46a8-8241-d0ff47adf533\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T00:40:27Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"12a38754-dc02-4ad3-b4da-a109e0fc5647\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:40:27Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:40:47Z)) }]), name: Some(\"brs-ip-10-10-125-43.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-125-43.ap-northeast-1.compute.internal\", uid: \"12a38754-dc02-4ad3-b4da-a109e0fc5647\" }]), resource_version: Some(\"758584389\"), self_link: None, uid: Some(\"e565a17a-dce6-4858-b913-f70c20727b9f\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T01:36:06Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"b65bbb3b-2646-4f85-911f-576e5c7e79a7\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:36:06Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:36:26Z)) }]), name: Some(\"brs-ip-10-10-152-12.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-152-12.ap-northeast-1.compute.internal\", uid: \"b65bbb3b-2646-4f85-911f-576e5c7e79a7\" }]), resource_version: Some(\"758618324\"), self_link: None, uid: Some(\"e4475fd9-c910-4b95-99e4-d3310802e2a8\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T01:46:55Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"4c1b64f1-0f66-4eb0-b575-d21178ae2ef4\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:46:55Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:47:15Z)) }]), name: Some(\"brs-ip-10-10-50-25.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-50-25.ap-northeast-1.compute.internal\", uid: \"4c1b64f1-0f66-4eb0-b575-d21178ae2ef4\" }]), resource_version: Some(\"758628025\"), self_link: None, uid: Some(\"eaff33e0-fb1d-4eba-9537-e53bccc0d9cf\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T00:35:05Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"a78ed789-e398-4597-8bc0-9fb2d0f3f9e1\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:35:05Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:35:25Z)) }]), name: Some(\"brs-ip-10-10-60-250.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-60-250.ap-northeast-1.compute.internal\", uid: \"a78ed789-e398-4597-8bc0-9fb2d0f3f9e1\" }]), resource_version: Some(\"758580152\"), self_link: None, uid: Some(\"18f25010-3c7f-4562-8f08-d3fc307419e6\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-11T09:21:51Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"d3f8deaa-a8bf-47e3-89c6-4b3a5ccfd5df\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-11T09:21:51Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-11T09:22:11Z)) }]), name: Some(\"brs-ip-10-10-140-10.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-140-10.ap-northeast-1.compute.internal\", uid: \"d3f8deaa-a8bf-47e3-89c6-4b3a5ccfd5df\" }]), resource_version: Some(\"750898787\"), self_link: None, uid: Some(\"b7801562-0758-486f-a651-522b745e96e2\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T01:41:27Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"baac4d42-802a-46b5-8bd8-5da49018c4b4\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:41:27Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:41:47Z)) }]), name: Some(\"brs-ip-10-10-99-185.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-99-185.ap-northeast-1.compute.internal\", uid: \"baac4d42-802a-46b5-8bd8-5da49018c4b4\" }]), resource_version: Some(\"758623288\"), self_link: None, uid: Some(\"7277b3a7-2f36-42e9-a508-c6dd08064b77\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T23:59:18Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"f800ec10-80a4-41ab-bfb7-4a0966710ce5\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T23:59:18Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T23:59:38Z)) }]), name: Some(\"brs-ip-10-10-103-221.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-103-221.ap-northeast-1.compute.internal\", uid: \"f800ec10-80a4-41ab-bfb7-4a0966710ce5\" }]), resource_version: Some(\"759367926\"), self_link: None, uid: Some(\"e27d9bbd-650c-437b-92e0-127baa9462a7\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T00:29:27Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"1a303426-16e2-4587-a7ee-a49b9e8654df\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:29:27Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:29:48Z)) }]), name: Some(\"brs-ip-10-10-36-216.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-36-216.ap-northeast-1.compute.internal\", uid: \"1a303426-16e2-4587-a7ee-a49b9e8654df\" }]), resource_version: Some(\"758575323\"), self_link: None, uid: Some(\"c8b16923-8baf-4949-8209-c42abf8091e2\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }]"}
{"v":0,"name":"brupop-controller","msg":"[SORT_SHADOWS - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605264828+00:00","target":"controller::controller","line":236,"file":"controller/src/controller.rs","associated_brs_name":"brs-ip-10-10-131-243.ap-northeast-1.compute.internal","elapsed_milliseconds":0,"shadows":"[BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T01:30:42Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"b24116e7-9e56-41a0-9a4d-5df1650ff6fe\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:30:42Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:31:03Z)) }]), name: Some(\"brs-ip-10-10-150-7.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-150-7.ap-northeast-1.compute.internal\", uid: \"b24116e7-9e56-41a0-9a4d-5df1650ff6fe\" }]), resource_version: Some(\"758613376\"), self_link: None, uid: Some(\"c6c3b44b-25ee-46a8-8241-d0ff47adf533\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T00:40:27Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"12a38754-dc02-4ad3-b4da-a109e0fc5647\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:40:27Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:40:47Z)) }]), name: Some(\"brs-ip-10-10-125-43.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-125-43.ap-northeast-1.compute.internal\", uid: \"12a38754-dc02-4ad3-b4da-a109e0fc5647\" }]), resource_version: Some(\"758584389\"), self_link: None, uid: Some(\"e565a17a-dce6-4858-b913-f70c20727b9f\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T01:36:06Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"b65bbb3b-2646-4f85-911f-576e5c7e79a7\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:36:06Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:36:26Z)) }]), name: Some(\"brs-ip-10-10-152-12.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-152-12.ap-northeast-1.compute.internal\", uid: \"b65bbb3b-2646-4f85-911f-576e5c7e79a7\" }]), resource_version: Some(\"758618324\"), self_link: None, uid: Some(\"e4475fd9-c910-4b95-99e4-d3310802e2a8\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T01:46:55Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"4c1b64f1-0f66-4eb0-b575-d21178ae2ef4\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:46:55Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:47:15Z)) }]), name: Some(\"brs-ip-10-10-50-25.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-50-25.ap-northeast-1.compute.internal\", uid: \"4c1b64f1-0f66-4eb0-b575-d21178ae2ef4\" }]), resource_version: Some(\"758628025\"), self_link: None, uid: Some(\"eaff33e0-fb1d-4eba-9537-e53bccc0d9cf\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T00:35:05Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"a78ed789-e398-4597-8bc0-9fb2d0f3f9e1\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:35:05Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:35:25Z)) }]), name: Some(\"brs-ip-10-10-60-250.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-60-250.ap-northeast-1.compute.internal\", uid: \"a78ed789-e398-4597-8bc0-9fb2d0f3f9e1\" }]), resource_version: Some(\"758580152\"), self_link: None, uid: Some(\"18f25010-3c7f-4562-8f08-d3fc307419e6\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-11T09:21:51Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"d3f8deaa-a8bf-47e3-89c6-4b3a5ccfd5df\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-11T09:21:51Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-11T09:22:11Z)) }]), name: Some(\"brs-ip-10-10-140-10.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-140-10.ap-northeast-1.compute.internal\", uid: \"d3f8deaa-a8bf-47e3-89c6-4b3a5ccfd5df\" }]), resource_version: Some(\"750898787\"), self_link: None, uid: Some(\"b7801562-0758-486f-a651-522b745e96e2\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T01:41:27Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"baac4d42-802a-46b5-8bd8-5da49018c4b4\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:41:27Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:41:47Z)) }]), name: Some(\"brs-ip-10-10-99-185.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-99-185.ap-northeast-1.compute.internal\", uid: \"baac4d42-802a-46b5-8bd8-5da49018c4b4\" }]), resource_version: Some(\"758623288\"), self_link: None, uid: Some(\"7277b3a7-2f36-42e9-a508-c6dd08064b77\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T23:59:18Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"f800ec10-80a4-41ab-bfb7-4a0966710ce5\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T23:59:18Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T23:59:38Z)) }]), name: Some(\"brs-ip-10-10-103-221.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-103-221.ap-northeast-1.compute.internal\", uid: \"f800ec10-80a4-41ab-bfb7-4a0966710ce5\" }]), resource_version: Some(\"759367926\"), self_link: None, uid: Some(\"e27d9bbd-650c-437b-92e0-127baa9462a7\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T00:29:27Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"1a303426-16e2-4587-a7ee-a49b9e8654df\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:29:27Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:29:48Z)) }]), name: Some(\"brs-ip-10-10-36-216.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-36-216.ap-northeast-1.compute.internal\", uid: \"1a303426-16e2-4587-a7ee-a49b9e8654df\" }]), resource_version: Some(\"758575323\"), self_link: None, uid: Some(\"c8b16923-8baf-4949-8209-c42abf8091e2\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }]"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605381223+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605395283+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605409544+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605422579+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605437846+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605448522+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605463969+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605475988+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605486983+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605499113+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605514270+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605525706+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605536596+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605549021+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605560518+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605572259+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605587855+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605599132+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605613923+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605623562+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[FIND_AND_UPDATE_READY_NODE - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605655580+00:00","target":"controller::controller","line":112,"file":"controller/src/controller.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[EMIT_METRICS - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605667811+00:00","target":"controller::controller","line":134,"file":"controller/src/controller.rs"}
{"v":0,"name":"brupop-controller","msg":"[FETCH_DATA - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605676298+00:00","target":"controller::controller","line":144,"file":"controller/src/controller.rs"}
{"v":0,"name":"brupop-controller","msg":"[FETCH_DATA - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605791216+00:00","target":"controller::controller","line":144,"file":"controller/src/controller.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[EMIT_METRICS - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:34.605804040+00:00","target":"controller::controller","line":134,"file":"controller/src/controller.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[ACTIVE_NODE_SET - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.606495616+00:00","target":"controller::controller","line":58,"file":"controller/src/controller.rs"}
{"v":0,"name":"brupop-controller","msg":"[ACTIVE_NODE_SET - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.606669265+00:00","target":"controller::controller","line":58,"file":"controller/src/controller.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[FIND_AND_UPDATE_READY_NODE - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.606736517+00:00","target":"controller::controller","line":112,"file":"controller/src/controller.rs"}
{"v":0,"name":"brupop-controller","msg":"[GET_ASSOCIATED_BOTTLEROCKETSHADOW_NAME - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.606821298+00:00","target":"controller::controller","line":218,"file":"controller/src/controller.rs"}
{"v":0,"name":"brupop-controller","msg":"[GET_ASSOCIATED_BOTTLEROCKETSHADOW_NAME - EVENT] Found associated bottlerocketshadow name.","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.606843409+00:00","target":"controller::controller","line":224,"file":"controller/src/controller.rs","associated_bottlerocketshadow_name":"\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\""}
{"v":0,"name":"brupop-controller","msg":"[GET_ASSOCIATED_BOTTLEROCKETSHADOW_NAME - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.606859534+00:00","target":"controller::controller","line":218,"file":"controller/src/controller.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[SORT_SHADOWS - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607201013+00:00","target":"controller::controller","line":236,"file":"controller/src/controller.rs","shadows":"[BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T01:30:42Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"b24116e7-9e56-41a0-9a4d-5df1650ff6fe\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:30:42Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:31:03Z)) }]), name: Some(\"brs-ip-10-10-150-7.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-150-7.ap-northeast-1.compute.internal\", uid: \"b24116e7-9e56-41a0-9a4d-5df1650ff6fe\" }]), resource_version: Some(\"758613376\"), self_link: None, uid: Some(\"c6c3b44b-25ee-46a8-8241-d0ff47adf533\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T00:40:27Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"12a38754-dc02-4ad3-b4da-a109e0fc5647\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:40:27Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:40:47Z)) }]), name: Some(\"brs-ip-10-10-125-43.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-125-43.ap-northeast-1.compute.internal\", uid: \"12a38754-dc02-4ad3-b4da-a109e0fc5647\" }]), resource_version: Some(\"758584389\"), self_link: None, uid: Some(\"e565a17a-dce6-4858-b913-f70c20727b9f\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T01:36:06Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"b65bbb3b-2646-4f85-911f-576e5c7e79a7\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:36:06Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:36:26Z)) }]), name: Some(\"brs-ip-10-10-152-12.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-152-12.ap-northeast-1.compute.internal\", uid: \"b65bbb3b-2646-4f85-911f-576e5c7e79a7\" }]), resource_version: Some(\"758618324\"), self_link: None, uid: Some(\"e4475fd9-c910-4b95-99e4-d3310802e2a8\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T01:46:55Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"4c1b64f1-0f66-4eb0-b575-d21178ae2ef4\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:46:55Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:47:15Z)) }]), name: Some(\"brs-ip-10-10-50-25.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-50-25.ap-northeast-1.compute.internal\", uid: \"4c1b64f1-0f66-4eb0-b575-d21178ae2ef4\" }]), resource_version: Some(\"758628025\"), self_link: None, uid: Some(\"eaff33e0-fb1d-4eba-9537-e53bccc0d9cf\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T00:35:05Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"a78ed789-e398-4597-8bc0-9fb2d0f3f9e1\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:35:05Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:35:25Z)) }]), name: Some(\"brs-ip-10-10-60-250.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-60-250.ap-northeast-1.compute.internal\", uid: \"a78ed789-e398-4597-8bc0-9fb2d0f3f9e1\" }]), resource_version: Some(\"758580152\"), self_link: None, uid: Some(\"18f25010-3c7f-4562-8f08-d3fc307419e6\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-11T09:21:51Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"d3f8deaa-a8bf-47e3-89c6-4b3a5ccfd5df\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-11T09:21:51Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-11T09:22:11Z)) }]), name: Some(\"brs-ip-10-10-140-10.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-140-10.ap-northeast-1.compute.internal\", uid: \"d3f8deaa-a8bf-47e3-89c6-4b3a5ccfd5df\" }]), resource_version: Some(\"750898787\"), self_link: None, uid: Some(\"b7801562-0758-486f-a651-522b745e96e2\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T01:41:27Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"baac4d42-802a-46b5-8bd8-5da49018c4b4\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:41:27Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:41:47Z)) }]), name: Some(\"brs-ip-10-10-99-185.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-99-185.ap-northeast-1.compute.internal\", uid: \"baac4d42-802a-46b5-8bd8-5da49018c4b4\" }]), resource_version: Some(\"758623288\"), self_link: None, uid: Some(\"7277b3a7-2f36-42e9-a508-c6dd08064b77\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T23:59:18Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"f800ec10-80a4-41ab-bfb7-4a0966710ce5\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T23:59:18Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T23:59:38Z)) }]), name: Some(\"brs-ip-10-10-103-221.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-103-221.ap-northeast-1.compute.internal\", uid: \"f800ec10-80a4-41ab-bfb7-4a0966710ce5\" }]), resource_version: Some(\"759367926\"), self_link: None, uid: Some(\"e27d9bbd-650c-437b-92e0-127baa9462a7\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T00:29:27Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"1a303426-16e2-4587-a7ee-a49b9e8654df\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:29:27Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:29:48Z)) }]), name: Some(\"brs-ip-10-10-36-216.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-36-216.ap-northeast-1.compute.internal\", uid: \"1a303426-16e2-4587-a7ee-a49b9e8654df\" }]), resource_version: Some(\"758575323\"), self_link: None, uid: Some(\"c8b16923-8baf-4949-8209-c42abf8091e2\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }]","associated_brs_name":"brs-ip-10-10-131-243.ap-northeast-1.compute.internal"}
{"v":0,"name":"brupop-controller","msg":"[SORT_SHADOWS - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607316250+00:00","target":"controller::controller","line":236,"file":"controller/src/controller.rs","elapsed_milliseconds":0,"shadows":"[BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T01:30:42Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"b24116e7-9e56-41a0-9a4d-5df1650ff6fe\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:30:42Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:31:03Z)) }]), name: Some(\"brs-ip-10-10-150-7.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-150-7.ap-northeast-1.compute.internal\", uid: \"b24116e7-9e56-41a0-9a4d-5df1650ff6fe\" }]), resource_version: Some(\"758613376\"), self_link: None, uid: Some(\"c6c3b44b-25ee-46a8-8241-d0ff47adf533\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T00:40:27Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"12a38754-dc02-4ad3-b4da-a109e0fc5647\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:40:27Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:40:47Z)) }]), name: Some(\"brs-ip-10-10-125-43.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-125-43.ap-northeast-1.compute.internal\", uid: \"12a38754-dc02-4ad3-b4da-a109e0fc5647\" }]), resource_version: Some(\"758584389\"), self_link: None, uid: Some(\"e565a17a-dce6-4858-b913-f70c20727b9f\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T01:36:06Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"b65bbb3b-2646-4f85-911f-576e5c7e79a7\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:36:06Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:36:26Z)) }]), name: Some(\"brs-ip-10-10-152-12.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-152-12.ap-northeast-1.compute.internal\", uid: \"b65bbb3b-2646-4f85-911f-576e5c7e79a7\" }]), resource_version: Some(\"758618324\"), self_link: None, uid: Some(\"e4475fd9-c910-4b95-99e4-d3310802e2a8\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T01:46:55Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"4c1b64f1-0f66-4eb0-b575-d21178ae2ef4\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:46:55Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:47:15Z)) }]), name: Some(\"brs-ip-10-10-50-25.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-50-25.ap-northeast-1.compute.internal\", uid: \"4c1b64f1-0f66-4eb0-b575-d21178ae2ef4\" }]), resource_version: Some(\"758628025\"), self_link: None, uid: Some(\"eaff33e0-fb1d-4eba-9537-e53bccc0d9cf\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T00:35:05Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"a78ed789-e398-4597-8bc0-9fb2d0f3f9e1\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:35:05Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:35:25Z)) }]), name: Some(\"brs-ip-10-10-60-250.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-60-250.ap-northeast-1.compute.internal\", uid: \"a78ed789-e398-4597-8bc0-9fb2d0f3f9e1\" }]), resource_version: Some(\"758580152\"), self_link: None, uid: Some(\"18f25010-3c7f-4562-8f08-d3fc307419e6\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-11T09:21:51Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"d3f8deaa-a8bf-47e3-89c6-4b3a5ccfd5df\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-11T09:21:51Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-11T09:22:11Z)) }]), name: Some(\"brs-ip-10-10-140-10.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-140-10.ap-northeast-1.compute.internal\", uid: \"d3f8deaa-a8bf-47e3-89c6-4b3a5ccfd5df\" }]), resource_version: Some(\"750898787\"), self_link: None, uid: Some(\"b7801562-0758-486f-a651-522b745e96e2\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T01:41:27Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"baac4d42-802a-46b5-8bd8-5da49018c4b4\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:41:27Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:41:47Z)) }]), name: Some(\"brs-ip-10-10-99-185.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-99-185.ap-northeast-1.compute.internal\", uid: \"baac4d42-802a-46b5-8bd8-5da49018c4b4\" }]), resource_version: Some(\"758623288\"), self_link: None, uid: Some(\"7277b3a7-2f36-42e9-a508-c6dd08064b77\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T23:59:18Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"f800ec10-80a4-41ab-bfb7-4a0966710ce5\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T23:59:18Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T23:59:38Z)) }]), name: Some(\"brs-ip-10-10-103-221.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-103-221.ap-northeast-1.compute.internal\", uid: \"f800ec10-80a4-41ab-bfb7-4a0966710ce5\" }]), resource_version: Some(\"759367926\"), self_link: None, uid: Some(\"e27d9bbd-650c-437b-92e0-127baa9462a7\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T00:29:27Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"1a303426-16e2-4587-a7ee-a49b9e8654df\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:29:27Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:29:48Z)) }]), name: Some(\"brs-ip-10-10-36-216.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-36-216.ap-northeast-1.compute.internal\", uid: \"1a303426-16e2-4587-a7ee-a49b9e8654df\" }]), resource_version: Some(\"758575323\"), self_link: None, uid: Some(\"c8b16923-8baf-4949-8209-c42abf8091e2\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }]","associated_brs_name":"brs-ip-10-10-131-243.ap-northeast-1.compute.internal"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607431918+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607445986+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607464471+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607478015+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607495215+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607509364+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607521761+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607533284+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607543341+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607555696+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607570524+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607580003+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607589139+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607600900+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607613698+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607626731+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607640853+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607650697+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607658769+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607667407+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[FIND_AND_UPDATE_READY_NODE - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607692481+00:00","target":"controller::controller","line":112,"file":"controller/src/controller.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[EMIT_METRICS - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607706065+00:00","target":"controller::controller","line":134,"file":"controller/src/controller.rs"}
{"v":0,"name":"brupop-controller","msg":"[FETCH_DATA - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607711679+00:00","target":"controller::controller","line":144,"file":"controller/src/controller.rs"}
{"v":0,"name":"brupop-controller","msg":"[FETCH_DATA - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607864115+00:00","target":"controller::controller","line":144,"file":"controller/src/controller.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[EMIT_METRICS - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:36.607876253+00:00","target":"controller::controller","line":134,"file":"controller/src/controller.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[ACTIVE_NODE_SET - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.608476165+00:00","target":"controller::controller","line":58,"file":"controller/src/controller.rs"}
{"v":0,"name":"brupop-controller","msg":"[ACTIVE_NODE_SET - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.608647189+00:00","target":"controller::controller","line":58,"file":"controller/src/controller.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[FIND_AND_UPDATE_READY_NODE - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.608695615+00:00","target":"controller::controller","line":112,"file":"controller/src/controller.rs"}
{"v":0,"name":"brupop-controller","msg":"[GET_ASSOCIATED_BOTTLEROCKETSHADOW_NAME - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.608779507+00:00","target":"controller::controller","line":218,"file":"controller/src/controller.rs"}
{"v":0,"name":"brupop-controller","msg":"[GET_ASSOCIATED_BOTTLEROCKETSHADOW_NAME - EVENT] Found associated bottlerocketshadow name.","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.608801352+00:00","target":"controller::controller","line":224,"file":"controller/src/controller.rs","associated_bottlerocketshadow_name":"\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\""}
{"v":0,"name":"brupop-controller","msg":"[GET_ASSOCIATED_BOTTLEROCKETSHADOW_NAME - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.608814163+00:00","target":"controller::controller","line":218,"file":"controller/src/controller.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[SORT_SHADOWS - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609147349+00:00","target":"controller::controller","line":236,"file":"controller/src/controller.rs","shadows":"[BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T01:30:42Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"b24116e7-9e56-41a0-9a4d-5df1650ff6fe\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:30:42Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:31:03Z)) }]), name: Some(\"brs-ip-10-10-150-7.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-150-7.ap-northeast-1.compute.internal\", uid: \"b24116e7-9e56-41a0-9a4d-5df1650ff6fe\" }]), resource_version: Some(\"758613376\"), self_link: None, uid: Some(\"c6c3b44b-25ee-46a8-8241-d0ff47adf533\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T00:40:27Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"12a38754-dc02-4ad3-b4da-a109e0fc5647\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:40:27Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:40:47Z)) }]), name: Some(\"brs-ip-10-10-125-43.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-125-43.ap-northeast-1.compute.internal\", uid: \"12a38754-dc02-4ad3-b4da-a109e0fc5647\" }]), resource_version: Some(\"758584389\"), self_link: None, uid: Some(\"e565a17a-dce6-4858-b913-f70c20727b9f\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T01:36:06Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"b65bbb3b-2646-4f85-911f-576e5c7e79a7\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:36:06Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:36:26Z)) }]), name: Some(\"brs-ip-10-10-152-12.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-152-12.ap-northeast-1.compute.internal\", uid: \"b65bbb3b-2646-4f85-911f-576e5c7e79a7\" }]), resource_version: Some(\"758618324\"), self_link: None, uid: Some(\"e4475fd9-c910-4b95-99e4-d3310802e2a8\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T01:46:55Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"4c1b64f1-0f66-4eb0-b575-d21178ae2ef4\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:46:55Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:47:15Z)) }]), name: Some(\"brs-ip-10-10-50-25.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-50-25.ap-northeast-1.compute.internal\", uid: \"4c1b64f1-0f66-4eb0-b575-d21178ae2ef4\" }]), resource_version: Some(\"758628025\"), self_link: None, uid: Some(\"eaff33e0-fb1d-4eba-9537-e53bccc0d9cf\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T00:35:05Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"a78ed789-e398-4597-8bc0-9fb2d0f3f9e1\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:35:05Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:35:25Z)) }]), name: Some(\"brs-ip-10-10-60-250.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-60-250.ap-northeast-1.compute.internal\", uid: \"a78ed789-e398-4597-8bc0-9fb2d0f3f9e1\" }]), resource_version: Some(\"758580152\"), self_link: None, uid: Some(\"18f25010-3c7f-4562-8f08-d3fc307419e6\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-11T09:21:51Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"d3f8deaa-a8bf-47e3-89c6-4b3a5ccfd5df\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-11T09:21:51Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-11T09:22:11Z)) }]), name: Some(\"brs-ip-10-10-140-10.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-140-10.ap-northeast-1.compute.internal\", uid: \"d3f8deaa-a8bf-47e3-89c6-4b3a5ccfd5df\" }]), resource_version: Some(\"750898787\"), self_link: None, uid: Some(\"b7801562-0758-486f-a651-522b745e96e2\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T01:41:27Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"baac4d42-802a-46b5-8bd8-5da49018c4b4\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:41:27Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:41:47Z)) }]), name: Some(\"brs-ip-10-10-99-185.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-99-185.ap-northeast-1.compute.internal\", uid: \"baac4d42-802a-46b5-8bd8-5da49018c4b4\" }]), resource_version: Some(\"758623288\"), self_link: None, uid: Some(\"7277b3a7-2f36-42e9-a508-c6dd08064b77\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T23:59:18Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"f800ec10-80a4-41ab-bfb7-4a0966710ce5\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T23:59:18Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T23:59:38Z)) }]), name: Some(\"brs-ip-10-10-103-221.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-103-221.ap-northeast-1.compute.internal\", uid: \"f800ec10-80a4-41ab-bfb7-4a0966710ce5\" }]), resource_version: Some(\"759367926\"), self_link: None, uid: Some(\"e27d9bbd-650c-437b-92e0-127baa9462a7\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T00:29:27Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"1a303426-16e2-4587-a7ee-a49b9e8654df\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:29:27Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:29:48Z)) }]), name: Some(\"brs-ip-10-10-36-216.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-36-216.ap-northeast-1.compute.internal\", uid: \"1a303426-16e2-4587-a7ee-a49b9e8654df\" }]), resource_version: Some(\"758575323\"), self_link: None, uid: Some(\"c8b16923-8baf-4949-8209-c42abf8091e2\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }]","associated_brs_name":"brs-ip-10-10-131-243.ap-northeast-1.compute.internal"}
{"v":0,"name":"brupop-controller","msg":"[SORT_SHADOWS - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609260100+00:00","target":"controller::controller","line":236,"file":"controller/src/controller.rs","shadows":"[BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T01:30:42Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"b24116e7-9e56-41a0-9a4d-5df1650ff6fe\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:30:42Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:31:03Z)) }]), name: Some(\"brs-ip-10-10-150-7.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-150-7.ap-northeast-1.compute.internal\", uid: \"b24116e7-9e56-41a0-9a4d-5df1650ff6fe\" }]), resource_version: Some(\"758613376\"), self_link: None, uid: Some(\"c6c3b44b-25ee-46a8-8241-d0ff47adf533\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T00:40:27Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"12a38754-dc02-4ad3-b4da-a109e0fc5647\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:40:27Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:40:47Z)) }]), name: Some(\"brs-ip-10-10-125-43.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-125-43.ap-northeast-1.compute.internal\", uid: \"12a38754-dc02-4ad3-b4da-a109e0fc5647\" }]), resource_version: Some(\"758584389\"), self_link: None, uid: Some(\"e565a17a-dce6-4858-b913-f70c20727b9f\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T01:36:06Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"b65bbb3b-2646-4f85-911f-576e5c7e79a7\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:36:06Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:36:26Z)) }]), name: Some(\"brs-ip-10-10-152-12.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-152-12.ap-northeast-1.compute.internal\", uid: \"b65bbb3b-2646-4f85-911f-576e5c7e79a7\" }]), resource_version: Some(\"758618324\"), self_link: None, uid: Some(\"e4475fd9-c910-4b95-99e4-d3310802e2a8\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T01:46:55Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"4c1b64f1-0f66-4eb0-b575-d21178ae2ef4\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:46:55Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:47:15Z)) }]), name: Some(\"brs-ip-10-10-50-25.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-50-25.ap-northeast-1.compute.internal\", uid: \"4c1b64f1-0f66-4eb0-b575-d21178ae2ef4\" }]), resource_version: Some(\"758628025\"), self_link: None, uid: Some(\"eaff33e0-fb1d-4eba-9537-e53bccc0d9cf\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T00:35:05Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"a78ed789-e398-4597-8bc0-9fb2d0f3f9e1\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:35:05Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:35:25Z)) }]), name: Some(\"brs-ip-10-10-60-250.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-60-250.ap-northeast-1.compute.internal\", uid: \"a78ed789-e398-4597-8bc0-9fb2d0f3f9e1\" }]), resource_version: Some(\"758580152\"), self_link: None, uid: Some(\"18f25010-3c7f-4562-8f08-d3fc307419e6\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-11T09:21:51Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"d3f8deaa-a8bf-47e3-89c6-4b3a5ccfd5df\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-11T09:21:51Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-11T09:22:11Z)) }]), name: Some(\"brs-ip-10-10-140-10.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-140-10.ap-northeast-1.compute.internal\", uid: \"d3f8deaa-a8bf-47e3-89c6-4b3a5ccfd5df\" }]), resource_version: Some(\"750898787\"), self_link: None, uid: Some(\"b7801562-0758-486f-a651-522b745e96e2\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-14T19:28:12Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"7e6de5b3-38fd-4448-8486-a9be640ea11f\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:12Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-14T19:28:33Z)) }]), name: Some(\"brs-ip-10-10-131-243.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-131-243.ap-northeast-1.compute.internal\", uid: \"7e6de5b3-38fd-4448-8486-a9be640ea11f\" }]), resource_version: Some(\"753626703\"), self_link: None, uid: Some(\"6a83433b-5011-4445-97ef-cf43b1b21cfc\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T01:41:27Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"baac4d42-802a-46b5-8bd8-5da49018c4b4\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:41:27Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T01:41:47Z)) }]), name: Some(\"brs-ip-10-10-99-185.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-99-185.ap-northeast-1.compute.internal\", uid: \"baac4d42-802a-46b5-8bd8-5da49018c4b4\" }]), resource_version: Some(\"758623288\"), self_link: None, uid: Some(\"7277b3a7-2f36-42e9-a508-c6dd08064b77\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T23:59:18Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"f800ec10-80a4-41ab-bfb7-4a0966710ce5\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T23:59:18Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T23:59:38Z)) }]), name: Some(\"brs-ip-10-10-103-221.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-103-221.ap-northeast-1.compute.internal\", uid: \"f800ec10-80a4-41ab-bfb7-4a0966710ce5\" }]), resource_version: Some(\"759367926\"), self_link: None, uid: Some(\"e27d9bbd-650c-437b-92e0-127baa9462a7\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-21T00:29:27Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:metadata\": Object({\"f:ownerReferences\": Object({\".\": Object({}), \"k:{\\\"uid\\\":\\\"1a303426-16e2-4587-a7ee-a49b9e8654df\\\"}\": Object({})})}), \"f:spec\": Object({\".\": Object({}), \"f:state\": Object({}), \"f:state_transition_timestamp\": Object({}), \"f:version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:29:27Z)) }, ManagedFieldsEntry { api_version: Some(\"brupop.bottlerocket.aws/v2\"), fields_type: Some(\"FieldsV1\"), fields_v1: Some(FieldsV1(Object({\"f:status\": Object({\".\": Object({}), \"f:crash_count\": Object({}), \"f:current_state\": Object({}), \"f:current_version\": Object({}), \"f:target_version\": Object({})})}))), manager: Some(\"unknown\"), operation: Some(\"Update\"), time: Some(Time(2022-09-21T00:29:48Z)) }]), name: Some(\"brs-ip-10-10-36-216.ap-northeast-1.compute.internal\"), namespace: Some(\"brupop-bottlerocket-aws\"), owner_references: Some([OwnerReference { api_version: \"v1\", block_owner_deletion: None, controller: None, kind: \"Node\", name: \"ip-10-10-36-216.ap-northeast-1.compute.internal\", uid: \"1a303426-16e2-4587-a7ee-a49b9e8654df\" }]), resource_version: Some(\"758575323\"), self_link: None, uid: Some(\"c8b16923-8baf-4949-8209-c42abf8091e2\") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: \"1.9.2\", target_version: \"1.9.2\", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }]","elapsed_milliseconds":0,"associated_brs_name":"brs-ip-10-10-131-243.ap-northeast-1.compute.internal"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609349502+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609375785+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609389165+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609395955+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609403595+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609413843+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609421222+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609430360+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609437223+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609442966+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609453706+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609462660+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609469961+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609478673+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609489780+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609500937+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609513463+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609522889+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609529757+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs"}
{"v":0,"name":"brupop-controller","msg":"[DETERMINE_NEXT_NODE_SPEC - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609535321+00:00","target":"controller::statemachine","line":13,"file":"controller/src/statemachine.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[FIND_AND_UPDATE_READY_NODE - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609557628+00:00","target":"controller::controller","line":112,"file":"controller/src/controller.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[EMIT_METRICS - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609570252+00:00","target":"controller::controller","line":134,"file":"controller/src/controller.rs"}
{"v":0,"name":"brupop-controller","msg":"[FETCH_DATA - START]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609578495+00:00","target":"controller::controller","line":144,"file":"controller/src/controller.rs"}
{"v":0,"name":"brupop-controller","msg":"[FETCH_DATA - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609694902+00:00","target":"controller::controller","line":144,"file":"controller/src/controller.rs","elapsed_milliseconds":0}
{"v":0,"name":"brupop-controller","msg":"[EMIT_METRICS - END]","level":30,"hostname":"brupop-controller-deployment-787f4cb7d6-sn8jr","pid":1,"time":"2022-09-22T08:08:38.609705810+00:00","target":"controller::controller","line":134,"file":"controller/src/controller.rs","elapsed_milliseconds":0}
I'm not seeing anything which would indicate it's querying for version 1.
The few logs with BottlerocketShadow objects seem to have api_version: Some(\"brupop.bottlerocket.aws/v2\").
I was able to reproduce this issue locally.
It turned out to be the controller was requesting v1 version for the custom resource:
[2022-09-22T22:26:09.547Z] INFO: brupop-controller/1 on brupop-controller-deployment-6484476846-vsbv5: [SORT_SHADOWS - END] (associated_brs_name=brs-ip-192-168-78-61.us-west-2.compute.internal,elapsed_milliseconds=0,file=controller/src/controller.rs,line=236,target=controller::controller)
shadows: [BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-22T17:40:24Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(1), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some("brupop.bottlerocket.aws/v1"), fields_type: Some("FieldsV1"), fields_v1: Some(FieldsV1(Object({"f:metadata": Object({"f:ownerReferences": Object({".": Object({}), "k:{\"uid\":\"c75d54e9-3b3f-4c0b-89db-fdc0d838d6ff\"}": Object({".": Object({}), "f:apiVersion": Object({}), "f:kind": Object({}), "f:name": Object({}), "f:uid": Object({})})})}), "f:spec": Object({".": Object({}), "f:state": Object({}), "f:state_transition_timestamp": Object({}), "f:version": Object({})}), "f:status": Object({".": Object({}), "f:current_state": Object({}), "f:current_version": Object({}), "f:target_version": Object({})})}))), manager: Some("unknown"), operation: Some("Update"), time: Some(Time(2022-09-22T17:40:44Z)) }]), name: Some("brs-ip-192-168-67-5.us-west-2.compute.internal"), namespace: Some("brupop-bottlerocket-aws"), owner_references: Some([OwnerReference { api_version: "v1", block_owner_deletion: None, controller: None, kind: "Node", name: "ip-192-168-67-5.us-west-2.compute.internal", uid: "c75d54e9-3b3f-4c0b-89db-fdc0d838d6ff" }]), resource_version: Some("16445291"), self_link: None, uid: Some("c9a3da4d-89e3-4b67-a77c-9283cae17bad") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: "1.9.2", target_version: "1.9.2", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }, BottlerocketShadow { metadata: ObjectMeta { annotations: None, cluster_name: None, creation_timestamp: Some(Time(2022-09-22T20:55:42Z)), deletion_grace_period_seconds: None, deletion_timestamp: None, finalizers: None, generate_name: None, generation: Some(6), labels: None, managed_fields: None, name: Some("brs-ip-192-168-78-61.us-west-2.compute.internal"), namespace: Some("brupop-bottlerocket-aws"), owner_references: Some([OwnerReference { api_version: "v1", block_owner_deletion: None, controller: None, kind: "Node", name: "ip-192-168-78-61.us-west-2.compute.internal", uid: "ec17f51a-4bc3-415b-97da-4103cba72c2b" }]), resource_version: Some("16480545"), self_link: None, uid: Some("ce1a0876-56a7-4840-8e97-cb34b520af74") }, spec: BottlerocketShadowSpec { state: Idle, state_transition_timestamp: None, version: None }, status: Some(BottlerocketShadowStatus { current_version: "1.9.2", target_version: "1.9.2", current_state: Idle, crash_count: 0, state_transition_failure_timestamp: None }) }]
The update process was acting normal on my side even with the error message, would you mind let me know if your cluster updates as expected? You could check by using:
➜ ~ kubectl get brs
NAME STATE VERSION TARGET STATE TARGET VERSION CRASH COUNT
brs-ip-192-168-67-5.us-west-2.compute.internal Idle 1.9.2 Idle <no value> 0
brs-ip-192-168-78-61.us-west-2.compute.internal Idle 1.9.2 Idle 0
The root cause is still not clear to me, we are actively working on this issue.
Actually, we updated the operator after all our nodes were updated to 1.9.2, so I can't confirm if it affects the updates or not.
We just detected an abnormal amount of logs coming from the operator and that's when we found the issue.
Thank you for investigating.