cloudstate
cloudstate copied to clipboard
Published
20 hours ago •
cloudstateio
cloudstateio
Native-image Cassandra smoke test is flaky
trafficstars
The native-image smoke test for Cassandra is often failing and too unreliable. This test will be disabled until we can address this.
Example logs for failed test
#!/bin/bash -eo pipefail
bin/run-java-eventsourced-shopping-cart-test.sh Cassandra --no-build
=== Running java-eventsourced-shopping-cart test ===
statefulstore.cloudstate.io/cassandra created
statefulservice.cloudstate.io/shopping-cart-cassandra created
Waiting for deployment to be created...
NAME READY UP-TO-DATE AVAILABLE AGE
shopping-cart-cassandra 0/1 1 0 1s
Waiting for deployment to be ready...
Waiting for deployment "shopping-cart-cassandra" rollout to finish: 0 of 1 updated replicas are available...
deployment "shopping-cart-cassandra" successfully rolled out
NAME READY UP-TO-DATE AVAILABLE AGE
shopping-cart-cassandra 1/1 1 1 8s
Scaling deployment...
statefulservice.cloudstate.io/shopping-cart-cassandra scaled
NAME READY UP-TO-DATE AVAILABLE AGE
shopping-cart-cassandra 1/3 1 1 8s
Waiting for deployment to be ready...
Waiting for deployment "shopping-cart-cassandra" rollout to finish: 1 of 3 updated replicas are available...
Waiting for deployment "shopping-cart-cassandra" rollout to finish: 2 of 3 updated replicas are available...
deployment "shopping-cart-cassandra" successfully rolled out
NAME READY UP-TO-DATE AVAILABLE AGE
shopping-cart-cassandra 3/3 3 3 23s
service/shopping-cart-cassandra-node-port exposed
Testing eventsourced shopping cart test1 ...
Expected '{"items":[]}'
But got 'The server was not able to produce a timely response to your request.
Please try again in a short while!'
Failed:
=== Operator logs ===
2021-02-02T00:03:21.517Z INFO setup using config {"file": "/etc/config/config.yaml"}
2021-02-02T00:03:22.172Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": "127.0.0.1:8080"}
2021-02-02T00:03:22.172Z INFO controller-runtime.webhook registering webhook {"path": "/inject-v1-pod"}
2021-02-02T00:03:22.172Z INFO setup starting manager
I0202 00:03:22.174217 1 leaderelection.go:242] attempting to acquire leader lease cloudstate-system/cloudstate-leader-election-helper...
2021-02-02T00:03:22.174Z INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
2021-02-02T00:03:22.175Z INFO controller-runtime.webhook.webhooks starting webhook server
2021-02-02T00:03:22.175Z INFO controller-runtime.certwatcher Updated current TLS certificate
2021-02-02T00:03:22.175Z INFO controller-runtime.webhook serving webhook server {"host": "", "port": 9443}
2021-02-02T00:03:22.217Z INFO controller-runtime.certwatcher Starting certificate watcher
I0202 00:03:22.222430 1 leaderelection.go:252] successfully acquired lease cloudstate-system/cloudstate-leader-election-helper
2021-02-02T00:03:22.222Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"ConfigMap","namespace":"cloudstate-system","name":"cloudstate-leader-election-helper","uid":"2f295404-3bbd-4c05-bea8-39e7d27c1e81","apiVersion":"v1","resourceVersion":"715"}, "reason": "LeaderElection", "message": "cloudstate-controller-manager-648c764f7b-84nsh_0288eb0a-1691-41b7-8a73-8a9ca7b6da8b became leader"}
2021-02-02T00:03:22.222Z INFO controller Starting EventSource {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "source": "kind source: /, Kind="}
2021-02-02T00:03:22.222Z INFO controller Starting EventSource {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulStore", "controller": "statefulstore", "source": "kind source: /, Kind="}
2021-02-02T00:03:22.323Z INFO controller Starting EventSource {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "source": "kind source: /, Kind="}
2021-02-02T00:03:22.323Z INFO controller Starting Controller {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulStore", "controller": "statefulstore"}
2021-02-02T00:03:22.517Z INFO controller Starting workers {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulStore", "controller": "statefulstore", "worker count": 1}
2021-02-02T00:03:22.517Z INFO controller Starting EventSource {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "source": "kind source: /, Kind="}
2021-02-02T00:03:22.618Z INFO controller Starting EventSource {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "source": "kind source: /, Kind="}
2021-02-02T00:03:22.718Z INFO controller Starting EventSource {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "source": "kind source: /, Kind="}
2021-02-02T00:03:22.819Z INFO controller Starting EventSource {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "source": "kind source: /, Kind="}
2021-02-02T00:03:22.919Z INFO controller Starting Controller {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice"}
2021-02-02T00:03:22.919Z INFO controller Starting workers {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "worker count": 1}
2021-02-02T00:03:27.020Z DEBUG controller-runtime.webhook.webhooks received request {"webhook": "/inject-v1-pod", "UID": "e5137699-440f-41eb-a033-6c4a242cab6c", "kind": "/v1, Kind=Pod", "resource": {"group":"","version":"v1","resource":"pods"}}
2021-02-02T00:03:27.026Z DEBUG controller-runtime.webhook.webhooks wrote response {"webhook": "/inject-v1-pod", "code": 200, "reason": "not a stateful service, not injecting", "UID": "e5137699-440f-41eb-a033-6c4a242cab6c", "allowed": true}
2021-02-02T00:04:45.218Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulStore", "controller": "statefulstore", "name": "cassandra", "namespace": "default"}
2021-02-02T00:04:45.218Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulStore", "controller": "statefulstore", "name": "cassandra", "namespace": "default"}
2021-02-02T00:04:45.543Z INFO controllers.StatefulService Updated status {"statefulservice": "default/shopping-cart-cassandra", "replicas": 0, "summary": "Unavailable"}
2021-02-02T00:04:45.548Z INFO controllers.StatefulService Created deployment {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:45.548Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:04:45.549Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 1, "status.updatedReplicas": 0, "status.availableReplicas": 0}
2021-02-02T00:04:45.558Z INFO controllers.StatefulService Updated status {"statefulservice": "default/shopping-cart-cassandra", "replicas": 0, "summary": "UpdateInProgress"}
2021-02-02T00:04:45.560Z DEBUG controller-runtime.webhook.webhooks received request {"webhook": "/inject-v1-pod", "UID": "574b962d-2e4d-4872-8cb0-16d964f7166c", "kind": "/v1, Kind=Pod", "resource": {"group":"","version":"v1","resource":"pods"}}
2021-02-02T00:04:45.617Z DEBUG controller-runtime.webhook.webhooks wrote response {"webhook": "/inject-v1-pod", "code": 200, "reason": "", "UID": "574b962d-2e4d-4872-8cb0-16d964f7166c", "allowed": true}
2021-02-02T00:04:45.618Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:45.618Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:04:45.619Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 1, "status.updatedReplicas": 0, "status.availableReplicas": 0}
2021-02-02T00:04:45.619Z DEBUG controllers.StatefulService No status update {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:45.621Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:45.621Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:04:45.646Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 1, "status.updatedReplicas": 1, "status.availableReplicas": 0}
2021-02-02T00:04:45.652Z INFO controllers.StatefulService Updated status {"statefulservice": "default/shopping-cart-cassandra", "replicas": 0, "summary": "Unavailable"}
2021-02-02T00:04:45.717Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:45.717Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:04:45.718Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 1, "status.updatedReplicas": 1, "status.availableReplicas": 0}
2021-02-02T00:04:45.718Z DEBUG controllers.StatefulService No status update {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:45.719Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:45.719Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:04:53.465Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 1, "status.updatedReplicas": 1, "status.availableReplicas": 1}
2021-02-02T00:04:53.517Z INFO controllers.StatefulService Updated status {"statefulservice": "default/shopping-cart-cassandra", "replicas": 1, "summary": "Ready"}
2021-02-02T00:04:53.519Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.519Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:04:53.520Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 1, "status.updatedReplicas": 1, "status.availableReplicas": 1}
2021-02-02T00:04:53.520Z DEBUG controllers.StatefulService No status update {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.522Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.522Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:04:53.676Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 1, "status.updatedReplicas": 1, "status.availableReplicas": 1}
2021-02-02T00:04:53.676Z DEBUG controllers.StatefulService No status update {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.679Z DEBUG controllers.StatefulService changed {"statefulservice": "default/shopping-cart-cassandra", "type": "deployment", "patchResult": "\nPatch: {\"metadata\":{\"annotations\":{\"banzaicloud.com/last-applied\":\"{\\\"metadata\\\":{\\\"labels\\\":{\\\"app.kubernetes.io/component\\\":\\\"user-function\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"cloudstate-operator\\\",\\\"app.kubernetes.io/name\\\":\\\"shopping-cart-cassandra\\\"},\\\"name\\\":\\\"shopping-cart-cassandra\\\",\\\"namespace\\\":\\\"default\\\",\\\"ownerReferences\\\":[{\\\"apiVersion\\\":\\\"cloudstate.io/v1alpha1\\\",\\\"blockOwnerDeletion\\\":true,\\\"controller\\\":true,\\\"kind\\\":\\\"StatefulService\\\",\\\"name\\\":\\\"shopping-cart-cassandra\\\",\\\"uid\\\":\\\"28abd19d-86cc-4692-bf2a-015a54be1cd1\\\"}]},\\\"spec\\\":{\\\"replicas\\\":3,\\\"selector\\\":{\\\"matchLabels\\\":{\\\"cloudstate.io/stateful-service\\\":\\\"shopping-cart-cassandra\\\"}},\\\"strategy\\\":{},\\\"template\\\":{\\\"metadata\\\":{\\\"annotations\\\":{\\\"cassandra.cloudstate.io/keyspace\\\":\\\"shoppingcart\\\",\\\"cloudstate.io/enabled\\\":\\\"true\\\",\\\"cloudstate.io/stateful-service-config\\\":\\\"ss-cfg-shopping-cart-cassandra\\\",\\\"cloudstate.io/stateful-store\\\":\\\"cassandra\\\"},\\\"labels\\\":{\\\"app.kubernetes.io/component\\\":\\\"user-function\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"cloudstate-operator\\\",\\\"app.kubernetes.io/name\\\":\\\"shopping-cart-cassandra\\\",\\\"cloudstate.io/stateful-service\\\":\\\"shopping-cart-cassandra\\\"}},\\\"spec\\\":{\\\"containers\\\":[{\\\"image\\\":\\\"cloudstateio/java-eventsourced-shopping-cart:latest\\\",\\\"imagePullPolicy\\\":\\\"Never\\\",\\\"name\\\":\\\"user-function\\\",\\\"resources\\\":{}}]}}},\\\"status\\\":{}}\"}},\"spec\":{\"replicas\":3}} \nCurrent: {\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{\"banzaicloud.com/last-applied\":\"{\\\"metadata\\\":{\\\"labels\\\":{\\\"app.kubernetes.io/component\\\":\\\"user-function\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"cloudstate-operator\\\",\\\"app.kubernetes.io/name\\\":\\\"shopping-cart-cassandra\\\"},\\\"name\\\":\\\"shopping-cart-cassandra\\\",\\\"namespace\\\":\\\"default\\\",\\\"ownerReferences\\\":[{\\\"apiVersion\\\":\\\"cloudstate.io/v1alpha1\\\",\\\"blockOwnerDeletion\\\":true,\\\"controller\\\":true,\\\"kind\\\":\\\"StatefulService\\\",\\\"name\\\":\\\"shopping-cart-cassandra\\\",\\\"uid\\\":\\\"28abd19d-86cc-4692-bf2a-015a54be1cd1\\\"}]},\\\"spec\\\":{\\\"replicas\\\":1,\\\"selector\\\":{\\\"matchLabels\\\":{\\\"cloudstate.io/stateful-service\\\":\\\"shopping-cart-cassandra\\\"}},\\\"strategy\\\":{},\\\"template\\\":{\\\"metadata\\\":{\\\"annotations\\\":{\\\"cassandra.cloudstate.io/keyspace\\\":\\\"shoppingcart\\\",\\\"cloudstate.io/enabled\\\":\\\"true\\\",\\\"cloudstate.io/stateful-service-config\\\":\\\"ss-cfg-shopping-cart-cassandra\\\",\\\"cloudstate.io/stateful-store\\\":\\\"cassandra\\\"},\\\"labels\\\":{\\\"app.kubernetes.io/component\\\":\\\"user-function\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"cloudstate-operator\\\",\\\"app.kubernetes.io/name\\\":\\\"shopping-cart-cassandra\\\",\\\"cloudstate.io/stateful-service\\\":\\\"shopping-cart-cassandra\\\"}},\\\"spec\\\":{\\\"containers\\\":[{\\\"image\\\":\\\"cloudstateio/java-eventsourced-shopping-cart:latest\\\",\\\"imagePullPolicy\\\":\\\"Never\\\",\\\"name\\\":\\\"user-function\\\",\\\"resources\\\":{}}]}}},\\\"status\\\":{}}\",\"deployment.kubernetes.io/revision\":\"1\"},\"creationTimestamp\":\"2021-02-02T00:04:45Z\",\"generation\":1,\"labels\":{\"app.kubernetes.io/component\":\"user-function\",\"app.kubernetes.io/managed-by\":\"cloudstate-operator\",\"app.kubernetes.io/name\":\"shopping-cart-cassandra\"},\"managedFields\":[{\"apiVersion\":\"apps/v1\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:banzaicloud.com/last-applied\":{}},\"f:labels\":{\".\":{},\"f:app.kubernetes.io/component\":{},\"f:app.kubernetes.io/managed-by\":{},\"f:app.kubernetes.io/name\":{}},\"f:ownerReferences\":{\".\":{},\"k:{\\\"uid\\\":\\\"28abd19d-86cc-4692-bf2a-015a54be1cd1\\\"}\":{\".\":{},\"f:apiVersion\":{},\"f:blockOwnerDeletion\":{},\"f:controller\":{},\"f:kind\":{},\"f:name\":{},\"f:uid\":{}}}},\"f:spec\":{\"f:progressDeadlineSeconds\":{},\"f:replicas\":{},\"f:revisionHistoryLimit\":{},\"f:selector\":{\"f:matchLabels\":{\".\":{},\"f:cloudstate.io/stateful-service\":{}}},\"f:strategy\":{\"f:rollingUpdate\":{\".\":{},\"f:maxSurge\":{},\"f:maxUnavailable\":{}},\"f:type\":{}},\"f:template\":{\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:cassandra.cloudstate.io/keyspace\":{},\"f:cloudstate.io/enabled\":{},\"f:cloudstate.io/stateful-service-config\":{},\"f:cloudstate.io/stateful-store\":{}},\"f:labels\":{\".\":{},\"f:app.kubernetes.io/component\":{},\"f:app.kubernetes.io/managed-by\":{},\"f:app.kubernetes.io/name\":{},\"f:cloudstate.io/stateful-service\":{}}},\"f:spec\":{\"f:containers\":{\"k:{\\\"name\\\":\\\"user-function\\\"}\":{\".\":{},\"f:image\":{},\"f:imagePullPolicy\":{},\"f:name\":{},\"f:resources\":{},\"f:terminationMessagePath\":{},\"f:terminationMessagePolicy\":{}}},\"f:dnsPolicy\":{},\"f:restartPolicy\":{},\"f:schedulerName\":{},\"f:securityContext\":{},\"f:terminationGracePeriodSeconds\":{}}}}},\"manager\":\"manager\",\"operation\":\"Update\",\"time\":\"2021-02-02T00:04:45Z\"},{\"apiVersion\":\"apps/v1\",\"fieldsType\":\"FieldsV1\",\"fieldsV1\":{\"f:metadata\":{\"f:annotations\":{\"f:deployment.kubernetes.io/revision\":{}}},\"f:status\":{\"f:availableReplicas\":{},\"f:conditions\":{\".\":{},\"k:{\\\"type\\\":\\\"Available\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:lastUpdateTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}},\"k:{\\\"type\\\":\\\"Progressing\\\"}\":{\".\":{},\"f:lastTransitionTime\":{},\"f:lastUpdateTime\":{},\"f:message\":{},\"f:reason\":{},\"f:status\":{},\"f:type\":{}}},\"f:observedGeneration\":{},\"f:readyReplicas\":{},\"f:replicas\":{},\"f:updatedReplicas\":{}}},\"manager\":\"kube-controller-manager\",\"operation\":\"Update\",\"time\":\"2021-02-02T00:04:53Z\"}],\"name\":\"shopping-cart-cassandra\",\"namespace\":\"default\",\"ownerReferences\":[{\"apiVersion\":\"cloudstate.io/v1alpha1\",\"blockOwnerDeletion\":true,\"controller\":true,\"kind\":\"StatefulService\",\"name\":\"shopping-cart-cassandra\",\"uid\":\"28abd19d-86cc-4692-bf2a-015a54be1cd1\"}],\"resourceVersion\":\"1014\",\"selfLink\":\"/apis/apps/v1/namespaces/default/deployments/shopping-cart-cassandra\",\"uid\":\"459e45de-a4a5-485a-ab9b-f296869812f3\"},\"spec\":{\"progressDeadlineSeconds\":600,\"replicas\":1,\"revisionHistoryLimit\":10,\"selector\":{\"matchLabels\":{\"cloudstate.io/stateful-service\":\"shopping-cart-cassandra\"}},\"strategy\":{\"rollingUpdate\":{\"maxSurge\":\"25%\",\"maxUnavailable\":\"25%\"},\"type\":\"RollingUpdate\"},\"template\":{\"metadata\":{\"annotations\":{\"cassandra.cloudstate.io/keyspace\":\"shoppingcart\",\"cloudstate.io/enabled\":\"true\",\"cloudstate.io/stateful-service-config\":\"ss-cfg-shopping-cart-cassandra\",\"cloudstate.io/stateful-store\":\"cassandra\"},\"labels\":{\"app.kubernetes.io/component\":\"user-function\",\"app.kubernetes.io/managed-by\":\"cloudstate-operator\",\"app.kubernetes.io/name\":\"shopping-cart-cassandra\",\"cloudstate.io/stateful-service\":\"shopping-cart-cassandra\"}},\"spec\":{\"containers\":[{\"image\":\"cloudstateio/java-eventsourced-shopping-cart:latest\",\"imagePullPolicy\":\"Never\",\"name\":\"user-function\",\"resources\":{},\"terminationMessagePath\":\"/dev/termination-log\",\"terminationMessagePolicy\":\"File\"}],\"dnsPolicy\":\"ClusterFirst\",\"restartPolicy\":\"Always\",\"schedulerName\":\"default-scheduler\",\"securityContext\":{},\"terminationGracePeriodSeconds\":30}}},\"status\":{\"availableReplicas\":1,\"conditions\":[{\"lastTransitionTime\":\"2021-02-02T00:04:53Z\",\"lastUpdateTime\":\"2021-02-02T00:04:53Z\",\"message\":\"Deployment has minimum availability.\",\"reason\":\"MinimumReplicasAvailable\",\"status\":\"True\",\"type\":\"Available\"},{\"lastTransitionTime\":\"2021-02-02T00:04:45Z\",\"lastUpdateTime\":\"2021-02-02T00:04:53Z\",\"message\":\"ReplicaSet \\\"shopping-cart-cassandra-65569f85cd\\\" has successfully progressed.\",\"reason\":\"NewReplicaSetAvailable\",\"status\":\"True\",\"type\":\"Progressing\"}],\"observedGeneration\":1,\"readyReplicas\":1,\"replicas\":1,\"updatedReplicas\":1}}\nModified: {\"metadata\":{\"annotations\":{\"banzaicloud.com/last-applied\":\"{\\\"metadata\\\":{\\\"labels\\\":{\\\"app.kubernetes.io/component\\\":\\\"user-function\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"cloudstate-operator\\\",\\\"app.kubernetes.io/name\\\":\\\"shopping-cart-cassandra\\\"},\\\"name\\\":\\\"shopping-cart-cassandra\\\",\\\"namespace\\\":\\\"default\\\",\\\"ownerReferences\\\":[{\\\"apiVersion\\\":\\\"cloudstate.io/v1alpha1\\\",\\\"blockOwnerDeletion\\\":true,\\\"controller\\\":true,\\\"kind\\\":\\\"StatefulService\\\",\\\"name\\\":\\\"shopping-cart-cassandra\\\",\\\"uid\\\":\\\"28abd19d-86cc-4692-bf2a-015a54be1cd1\\\"}]},\\\"spec\\\":{\\\"replicas\\\":3,\\\"selector\\\":{\\\"matchLabels\\\":{\\\"cloudstate.io/stateful-service\\\":\\\"shopping-cart-cassandra\\\"}},\\\"strategy\\\":{},\\\"template\\\":{\\\"metadata\\\":{\\\"annotations\\\":{\\\"cassandra.cloudstate.io/keyspace\\\":\\\"shoppingcart\\\",\\\"cloudstate.io/enabled\\\":\\\"true\\\",\\\"cloudstate.io/stateful-service-config\\\":\\\"ss-cfg-shopping-cart-cassandra\\\",\\\"cloudstate.io/stateful-store\\\":\\\"cassandra\\\"},\\\"labels\\\":{\\\"app.kubernetes.io/component\\\":\\\"user-function\\\",\\\"app.kubernetes.io/managed-by\\\":\\\"cloudstate-operator\\\",\\\"app.kubernetes.io/name\\\":\\\"shopping-cart-cassandra\\\",\\\"cloudstate.io/stateful-service\\\":\\\"shopping-cart-cassandra\\\"}},\\\"spec\\\":{\\\"containers\\\":[{\\\"image\\\":\\\"cloudstateio/java-eventsourced-shopping-cart:latest\\\",\\\"imagePullPolicy\\\":\\\"Never\\\",\\\"name\\\":\\\"user-function\\\",\\\"resources\\\":{}}]}}},\\\"status\\\":{}}\"},\"labels\":{\"app.kubernetes.io/component\":\"user-function\",\"app.kubernetes.io/managed-by\":\"cloudstate-operator\",\"app.kubernetes.io/name\":\"shopping-cart-cassandra\"},\"name\":\"shopping-cart-cassandra\",\"namespace\":\"default\",\"ownerReferences\":[{\"apiVersion\":\"cloudstate.io/v1alpha1\",\"blockOwnerDeletion\":true,\"controller\":true,\"kind\":\"StatefulService\",\"name\":\"shopping-cart-cassandra\",\"uid\":\"28abd19d-86cc-4692-bf2a-015a54be1cd1\"}]},\"spec\":{\"replicas\":3,\"selector\":{\"matchLabels\":{\"cloudstate.io/stateful-service\":\"shopping-cart-cassandra\"}},\"strategy\":{},\"template\":{\"metadata\":{\"annotations\":{\"cassandra.cloudstate.io/keyspace\":\"shoppingcart\",\"cloudstate.io/enabled\":\"true\",\"cloudstate.io/stateful-service-config\":\"ss-cfg-shopping-cart-cassandra\",\"cloudstate.io/stateful-store\":\"cassandra\"},\"labels\":{\"app.kubernetes.io/component\":\"user-function\",\"app.kubernetes.io/managed-by\":\"cloudstate-operator\",\"app.kubernetes.io/name\":\"shopping-cart-cassandra\",\"cloudstate.io/stateful-service\":\"shopping-cart-cassandra\"}},\"spec\":{\"containers\":[{\"image\":\"cloudstateio/java-eventsourced-shopping-cart:latest\",\"imagePullPolicy\":\"Never\",\"name\":\"user-function\",\"resources\":{}}]}}},\"status\":{}}\nOriginal: {\"metadata\":{\"labels\":{\"app.kubernetes.io/component\":\"user-function\",\"app.kubernetes.io/managed-by\":\"cloudstate-operator\",\"app.kubernetes.io/name\":\"shopping-cart-cassandra\"},\"name\":\"shopping-cart-cassandra\",\"namespace\":\"default\",\"ownerReferences\":[{\"apiVersion\":\"cloudstate.io/v1alpha1\",\"blockOwnerDeletion\":true,\"controller\":true,\"kind\":\"StatefulService\",\"name\":\"shopping-cart-cassandra\",\"uid\":\"28abd19d-86cc-4692-bf2a-015a54be1cd1\"}]},\"spec\":{\"replicas\":1,\"selector\":{\"matchLabels\":{\"cloudstate.io/stateful-service\":\"shopping-cart-cassandra\"}},\"strategy\":{},\"template\":{\"metadata\":{\"annotations\":{\"cassandra.cloudstate.io/keyspace\":\"shoppingcart\",\"cloudstate.io/enabled\":\"true\",\"cloudstate.io/stateful-service-config\":\"ss-cfg-shopping-cart-cassandra\",\"cloudstate.io/stateful-store\":\"cassandra\"},\"labels\":{\"app.kubernetes.io/component\":\"user-function\",\"app.kubernetes.io/managed-by\":\"cloudstate-operator\",\"app.kubernetes.io/name\":\"shopping-cart-cassandra\",\"cloudstate.io/stateful-service\":\"shopping-cart-cassandra\"}},\"spec\":{\"containers\":[{\"image\":\"cloudstateio/java-eventsourced-shopping-cart:latest\",\"imagePullPolicy\":\"Never\",\"name\":\"user-function\",\"resources\":{}}]}}},\"status\":{}}\n"}
2021-02-02T00:04:53.684Z INFO controllers.StatefulService Updated deployment {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.684Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:04:53.685Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 1, "status.availableReplicas": 1}
2021-02-02T00:04:53.702Z INFO controllers.StatefulService Updated status {"statefulservice": "default/shopping-cart-cassandra", "replicas": 1, "summary": "UpdateInProgress"}
2021-02-02T00:04:53.704Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.704Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:04:53.720Z DEBUG controller-runtime.webhook.webhooks received request {"webhook": "/inject-v1-pod", "UID": "3b3c3740-345a-4a90-84fc-2e890ad2cda8", "kind": "/v1, Kind=Pod", "resource": {"group":"","version":"v1","resource":"pods"}}
2021-02-02T00:04:53.721Z DEBUG controller-runtime.webhook.webhooks wrote response {"webhook": "/inject-v1-pod", "code": 200, "reason": "", "UID": "3b3c3740-345a-4a90-84fc-2e890ad2cda8", "allowed": true}
2021-02-02T00:04:53.723Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 1, "status.availableReplicas": 1}
2021-02-02T00:04:53.723Z DEBUG controllers.StatefulService No status update {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.726Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.726Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:04:53.727Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 1, "status.availableReplicas": 1}
2021-02-02T00:04:53.729Z DEBUG controllers.StatefulService No status update {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.732Z DEBUG controller-runtime.webhook.webhooks received request {"webhook": "/inject-v1-pod", "UID": "54b8cc51-9755-4154-ab50-18ca941a8a7e", "kind": "/v1, Kind=Pod", "resource": {"group":"","version":"v1","resource":"pods"}}
2021-02-02T00:04:53.817Z DEBUG controller-runtime.webhook.webhooks wrote response {"webhook": "/inject-v1-pod", "code": 200, "reason": "", "UID": "54b8cc51-9755-4154-ab50-18ca941a8a7e", "allowed": true}
2021-02-02T00:04:53.819Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.819Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:04:53.854Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 3, "status.availableReplicas": 1}
2021-02-02T00:04:53.919Z INFO controllers.StatefulService Updated status {"statefulservice": "default/shopping-cart-cassandra", "replicas": 1, "summary": "PartiallyReady"}
2021-02-02T00:04:53.920Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.920Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:04:53.921Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 3, "status.availableReplicas": 1}
2021-02-02T00:04:53.921Z DEBUG controllers.StatefulService No status update {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.923Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:04:53.923Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:05:00.926Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 3, "status.availableReplicas": 1}
2021-02-02T00:05:00.926Z DEBUG controllers.StatefulService No status update {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:05:00.928Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:05:00.928Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:05:08.418Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 3, "status.availableReplicas": 2}
2021-02-02T00:05:08.424Z INFO controllers.StatefulService Updated status {"statefulservice": "default/shopping-cart-cassandra", "replicas": 2, "summary": "PartiallyReady"}
2021-02-02T00:05:08.425Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:05:08.425Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:05:08.428Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 3, "status.availableReplicas": 2}
2021-02-02T00:05:08.428Z DEBUG controllers.StatefulService No status update {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:05:08.429Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:05:08.429Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:05:08.607Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 3, "status.availableReplicas": 3}
2021-02-02T00:05:08.614Z INFO controllers.StatefulService Updated status {"statefulservice": "default/shopping-cart-cassandra", "replicas": 3, "summary": "Ready"}
2021-02-02T00:05:08.615Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:05:08.615Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
2021-02-02T00:05:08.616Z DEBUG controllers.StatefulService Inspecting status {"statefulservice": "default/shopping-cart-cassandra", "spec.replicas": 3, "status.updatedReplicas": 3, "status.availableReplicas": 3}
2021-02-02T00:05:08.616Z DEBUG controllers.StatefulService No status update {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:05:08.617Z DEBUG controllers.StatefulService No change in deployment {"statefulservice": "default/shopping-cart-cassandra"}
2021-02-02T00:05:08.617Z DEBUG controller Successfully Reconciled {"reconcilerGroup": "cloudstate.io", "reconcilerKind": "StatefulService", "controller": "statefulservice", "name": "shopping-cart-cassandra", "namespace": "default"}
=== Deployment description ===
Name: shopping-cart-cassandra
Namespace: default
CreationTimestamp: Tue, 02 Feb 2021 00:04:45 +0000
Labels: app.kubernetes.io/component=user-function
app.kubernetes.io/managed-by=cloudstate-operator
app.kubernetes.io/name=shopping-cart-cassandra
Annotations: banzaicloud.com/last-applied:
{"metadata":{"labels":{"app.kubernetes.io/component":"user-function","app.kubernetes.io/managed-by":"cloudstate-operator","app.kubernetes....
deployment.kubernetes.io/revision: 1
Selector: cloudstate.io/stateful-service=shopping-cart-cassandra
Replicas: 3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app.kubernetes.io/component=user-function
app.kubernetes.io/managed-by=cloudstate-operator
app.kubernetes.io/name=shopping-cart-cassandra
cloudstate.io/stateful-service=shopping-cart-cassandra
Annotations: cassandra.cloudstate.io/keyspace: shoppingcart
cloudstate.io/enabled: true
cloudstate.io/stateful-service-config: ss-cfg-shopping-cart-cassandra
cloudstate.io/stateful-store: cassandra
Containers:
user-function:
Image: cloudstateio/java-eventsourced-shopping-cart:latest
Port: <none>
Host Port: <none>
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Progressing True NewReplicaSetAvailable
Available True MinimumReplicasAvailable
OldReplicaSets: <none>
NewReplicaSet: shopping-cart-cassandra-65569f85cd (3/3 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 44s deployment-controller Scaled up replica set shopping-cart-cassandra-65569f85cd to 1
Normal ScalingReplicaSet 36s deployment-controller Scaled up replica set shopping-cart-cassandra-65569f85cd to 3
=== Pods description ===
Name: cassandra-0
Namespace: default
Priority: 0
Node: default-235cd9d9-df1d-4cc5-b459-28892cef4871/10.128.1.60
Start Time: Tue, 02 Feb 2021 00:03:27 +0000
Labels: app=cassandra
chart=cassandra-5.6.1
controller-revision-hash=cassandra-67cf96f4b8
heritage=Helm
release=cassandra
statefulset.kubernetes.io/pod-name=cassandra-0
Annotations: <none>
Status: Running
IP: 172.17.0.7
IPs:
IP: 172.17.0.7
Controlled By: StatefulSet/cassandra
Containers:
cassandra:
Container ID: docker://ef7252fa690d037a6a83ee7c8334e33256c2ad57a742fb1c4eda6d9e5d39b58f
Image: docker.io/bitnami/cassandra:3.11.7-debian-10-r4
Image ID: docker-pullable://bitnami/cassandra@sha256:c864afbe56596553c72a3babf9c72158e89241a7e786eddbc2fa2551b56403ef
Ports: 7000/TCP, 7001/TCP, 7199/TCP, 9042/TCP, 9160/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
Command:
bash
-ec
if [[ $HOSTNAME =~ (.*)-0$ ]]; then
echo "Setting node as password seeder"
export CASSANDRA_PASSWORD_SEEDER=yes
else
# Only node 0 will execute the startup initdb scripts
export CASSANDRA_IGNORE_INITDB_SCRIPTS=1
fi
/entrypoint.sh /run.sh
State: Running
Started: Tue, 02 Feb 2021 00:03:40 +0000
Ready: True
Restart Count: 0
Liveness: exec [/bin/sh -c nodetool status] delay=60s timeout=5s period=30s #success=1 #failure=5
Readiness: exec [/bin/sh -c nodetool status | grep -E "^UN\s+${POD_IP}"] delay=60s timeout=5s period=10s #success=1 #failure=5
Environment:
BITNAMI_DEBUG: false
CASSANDRA_CLUSTER_NAME: cassandra
CASSANDRA_SEEDS: cassandra-0.cassandra-headless.default.svc.cluster.local
CASSANDRA_PASSWORD: <set to the key 'cassandra-password' in secret 'cassandra'> Optional: false
POD_IP: (v1:status.podIP)
CASSANDRA_USER: cassandra
CASSANDRA_NUM_TOKENS: 256
CASSANDRA_DATACENTER: dc1
CASSANDRA_ENDPOINT_SNITCH: SimpleSnitch
CASSANDRA_ENDPOINT_SNITCH: SimpleSnitch
CASSANDRA_RACK: rack1
CASSANDRA_ENABLE_RPC: true
Mounts:
/bitnami/cassandra from data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-wd7lz (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
default-token-wd7lz:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-wd7lz
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m2s default-scheduler Successfully assigned default/cassandra-0 to default-235cd9d9-df1d-4cc5-b459-28892cef4871
Normal Pulling 2m2s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Pulling image "docker.io/bitnami/cassandra:3.11.7-debian-10-r4"
Normal Pulled 109s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Successfully pulled image "docker.io/bitnami/cassandra:3.11.7-debian-10-r4"
Normal Created 109s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Created container cassandra
Normal Started 109s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Started container cassandra
Name: shopping-cart-cassandra-65569f85cd-dxpbr
Namespace: default
Priority: 0
Node: default-235cd9d9-df1d-4cc5-b459-28892cef4871/10.128.1.60
Start Time: Tue, 02 Feb 2021 00:04:53 +0000
Labels: app.kubernetes.io/component=user-function
app.kubernetes.io/managed-by=cloudstate-operator
app.kubernetes.io/name=shopping-cart-cassandra
cloudstate.io/stateful-service=shopping-cart-cassandra
pod-template-hash=65569f85cd
Annotations: cassandra.cloudstate.io/keyspace: shoppingcart
cloudstate.io/enabled: true
cloudstate.io/stateful-service-config: ss-cfg-shopping-cart-cassandra
cloudstate.io/stateful-store: cassandra
traffic.sidecar.istio.io/excludeInboundPorts: 2552,8558
traffic.sidecar.istio.io/excludeOutboundPorts: 2552,8558
Status: Running
IP: 172.17.0.10
IPs:
IP: 172.17.0.10
Controlled By: ReplicaSet/shopping-cart-cassandra-65569f85cd
Containers:
user-function:
Container ID: docker://062c42b378755454c4ca7be16a1f818bf62cfca5203bce8330133b9ac75dc40a
Image: cloudstateio/java-eventsourced-shopping-cart:latest
Image ID: docker://sha256:3df7437cc0f3bce3c4a5c9ff7293a884ce2e61a29cadbee171a849f204c5c35f
Port: 8080/TCP
Host Port: 0/TCP
State: Running
Started: Tue, 02 Feb 2021 00:04:54 +0000
Ready: True
Restart Count: 0
Environment:
PORT: 8080
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-wd7lz (ro)
cloudstate-sidecar:
Container ID: docker://4b0f0cc559a8517acd98ab2309470d97987af5a41d68a942e99fa910ddc1b213
Image: cloudstateio/cloudstate-proxy-native-cassandra:latest
Image ID: docker://sha256:0ba59b5fc83c69f703d7703cb92630c031944712d61d91efd14ec6056f249559
Ports: 8013/TCP, 9099/TCP
Host Ports: 0/TCP, 0/TCP
State: Running
Started: Tue, 02 Feb 2021 00:04:54 +0000
Ready: True
Restart Count: 0
Limits:
memory: 512Mi
Requests:
cpu: 400m
memory: 512Mi
Liveness: http-get http://:8558/alive delay=0s timeout=1s period=10s #success=1 #failure=5
Readiness: http-get http://:8558/ready delay=0s timeout=1s period=10s #success=1 #failure=5
Environment:
USER_FUNCTION_PORT: 8080
METRICS_PORT: 9099
REMOTING_PORT: 2552
MANAGEMENT_PORT: 8558
HTTP_PORT: 8013
SELECTOR_LABEL: cloudstate.io/stateful-service
SELECTOR_LABEL_VALUE: shopping-cart-cassandra
REQUIRED_CONTACT_POINT_NR: 1
CASSANDRA_CONTACT_POINTS: cassandra.default.svc.cluster.local
CASSANDRA_KEYSPACE: shoppingcart
CASSANDRA_USERNAME: <set to the key 'username' in secret 'cassandra-credentials'> Optional: false
CASSANDRA_PASSWORD: <set to the key 'password' in secret 'cassandra-credentials'> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-wd7lz (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-wd7lz:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-wd7lz
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 36s default-scheduler Successfully assigned default/shopping-cart-cassandra-65569f85cd-dxpbr to default-235cd9d9-df1d-4cc5-b459-28892cef4871
Normal Pulled 35s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Container image "cloudstateio/java-eventsourced-shopping-cart:latest" already present on machine
Normal Created 35s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Created container user-function
Normal Started 35s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Started container user-function
Normal Pulled 35s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Container image "cloudstateio/cloudstate-proxy-native-cassandra:latest" already present on machine
Normal Created 35s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Created container cloudstate-sidecar
Normal Started 35s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Started container cloudstate-sidecar
Warning Unhealthy 31s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Readiness probe failed: HTTP probe failed with statuscode: 500
Name: shopping-cart-cassandra-65569f85cd-g4vq5
Namespace: default
Priority: 0
Node: default-235cd9d9-df1d-4cc5-b459-28892cef4871/10.128.1.60
Start Time: Tue, 02 Feb 2021 00:04:45 +0000
Labels: app.kubernetes.io/component=user-function
app.kubernetes.io/managed-by=cloudstate-operator
app.kubernetes.io/name=shopping-cart-cassandra
cloudstate.io/stateful-service=shopping-cart-cassandra
pod-template-hash=65569f85cd
Annotations: cassandra.cloudstate.io/keyspace: shoppingcart
cloudstate.io/enabled: true
cloudstate.io/stateful-service-config: ss-cfg-shopping-cart-cassandra
cloudstate.io/stateful-store: cassandra
traffic.sidecar.istio.io/excludeInboundPorts: 2552,8558
traffic.sidecar.istio.io/excludeOutboundPorts: 2552,8558
Status: Running
IP: 172.17.0.8
IPs:
IP: 172.17.0.8
Controlled By: ReplicaSet/shopping-cart-cassandra-65569f85cd
Containers:
user-function:
Container ID: docker://b238c07e01234ef592fd85ca9f08d3902bc447c222911429e09a8044991461cc
Image: cloudstateio/java-eventsourced-shopping-cart:latest
Image ID: docker://sha256:3df7437cc0f3bce3c4a5c9ff7293a884ce2e61a29cadbee171a849f204c5c35f
Port: 8080/TCP
Host Port: 0/TCP
State: Running
Started: Tue, 02 Feb 2021 00:04:46 +0000
Ready: True
Restart Count: 0
Environment:
PORT: 8080
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-wd7lz (ro)
cloudstate-sidecar:
Container ID: docker://a97aca57bb76b2fe1e9913d5879dddab0f108e8fd0cee85933780d108bcffd7b
Image: cloudstateio/cloudstate-proxy-native-cassandra:latest
Image ID: docker://sha256:0ba59b5fc83c69f703d7703cb92630c031944712d61d91efd14ec6056f249559
Ports: 8013/TCP, 9099/TCP
Host Ports: 0/TCP, 0/TCP
State: Running
Started: Tue, 02 Feb 2021 00:04:46 +0000
Ready: True
Restart Count: 0
Limits:
memory: 512Mi
Requests:
cpu: 400m
memory: 512Mi
Liveness: http-get http://:8558/alive delay=0s timeout=1s period=10s #success=1 #failure=5
Readiness: http-get http://:8558/ready delay=0s timeout=1s period=10s #success=1 #failure=5
Environment:
USER_FUNCTION_PORT: 8080
METRICS_PORT: 9099
REMOTING_PORT: 2552
MANAGEMENT_PORT: 8558
HTTP_PORT: 8013
SELECTOR_LABEL: cloudstate.io/stateful-service
SELECTOR_LABEL_VALUE: shopping-cart-cassandra
REQUIRED_CONTACT_POINT_NR: 1
CASSANDRA_CONTACT_POINTS: cassandra.default.svc.cluster.local
CASSANDRA_KEYSPACE: shoppingcart
CASSANDRA_USERNAME: <set to the key 'username' in secret 'cassandra-credentials'> Optional: false
CASSANDRA_PASSWORD: <set to the key 'password' in secret 'cassandra-credentials'> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-wd7lz (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-wd7lz:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-wd7lz
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 44s default-scheduler Successfully assigned default/shopping-cart-cassandra-65569f85cd-g4vq5 to default-235cd9d9-df1d-4cc5-b459-28892cef4871
Normal Pulled 43s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Container image "cloudstateio/java-eventsourced-shopping-cart:latest" already present on machine
Normal Created 43s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Created container user-function
Normal Started 43s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Started container user-function
Normal Pulled 43s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Container image "cloudstateio/cloudstate-proxy-native-cassandra:latest" already present on machine
Normal Created 43s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Created container cloudstate-sidecar
Normal Started 43s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Started container cloudstate-sidecar
Name: shopping-cart-cassandra-65569f85cd-q6jd5
Namespace: default
Priority: 0
Node: default-235cd9d9-df1d-4cc5-b459-28892cef4871/10.128.1.60
Start Time: Tue, 02 Feb 2021 00:04:53 +0000
Labels: app.kubernetes.io/component=user-function
app.kubernetes.io/managed-by=cloudstate-operator
app.kubernetes.io/name=shopping-cart-cassandra
cloudstate.io/stateful-service=shopping-cart-cassandra
pod-template-hash=65569f85cd
Annotations: cassandra.cloudstate.io/keyspace: shoppingcart
cloudstate.io/enabled: true
cloudstate.io/stateful-service-config: ss-cfg-shopping-cart-cassandra
cloudstate.io/stateful-store: cassandra
traffic.sidecar.istio.io/excludeInboundPorts: 2552,8558
traffic.sidecar.istio.io/excludeOutboundPorts: 2552,8558
Status: Running
IP: 172.17.0.9
IPs:
IP: 172.17.0.9
Controlled By: ReplicaSet/shopping-cart-cassandra-65569f85cd
Containers:
user-function:
Container ID: docker://2f1dc02b9a6254404e8bcce1489d2c4c7b02ea01f3fb0c7729922cdfbf68ce7e
Image: cloudstateio/java-eventsourced-shopping-cart:latest
Image ID: docker://sha256:3df7437cc0f3bce3c4a5c9ff7293a884ce2e61a29cadbee171a849f204c5c35f
Port: 8080/TCP
Host Port: 0/TCP
State: Running
Started: Tue, 02 Feb 2021 00:04:54 +0000
Ready: True
Restart Count: 0
Environment:
PORT: 8080
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-wd7lz (ro)
cloudstate-sidecar:
Container ID: docker://b30cac0bd4cd0027d7773b3c401900438232779e633ae2f3cac54d0f3cd040c5
Image: cloudstateio/cloudstate-proxy-native-cassandra:latest
Image ID: docker://sha256:0ba59b5fc83c69f703d7703cb92630c031944712d61d91efd14ec6056f249559
Ports: 8013/TCP, 9099/TCP
Host Ports: 0/TCP, 0/TCP
State: Running
Started: Tue, 02 Feb 2021 00:04:54 +0000
Ready: True
Restart Count: 0
Limits:
memory: 512Mi
Requests:
cpu: 400m
memory: 512Mi
Liveness: http-get http://:8558/alive delay=0s timeout=1s period=10s #success=1 #failure=5
Readiness: http-get http://:8558/ready delay=0s timeout=1s period=10s #success=1 #failure=5
Environment:
USER_FUNCTION_PORT: 8080
METRICS_PORT: 9099
REMOTING_PORT: 2552
MANAGEMENT_PORT: 8558
HTTP_PORT: 8013
SELECTOR_LABEL: cloudstate.io/stateful-service
SELECTOR_LABEL_VALUE: shopping-cart-cassandra
REQUIRED_CONTACT_POINT_NR: 1
CASSANDRA_CONTACT_POINTS: cassandra.default.svc.cluster.local
CASSANDRA_KEYSPACE: shoppingcart
CASSANDRA_USERNAME: <set to the key 'username' in secret 'cassandra-credentials'> Optional: false
CASSANDRA_PASSWORD: <set to the key 'password' in secret 'cassandra-credentials'> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-wd7lz (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-wd7lz:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-wd7lz
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 36s default-scheduler Successfully assigned default/shopping-cart-cassandra-65569f85cd-q6jd5 to default-235cd9d9-df1d-4cc5-b459-28892cef4871
Normal Pulled 35s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Container image "cloudstateio/java-eventsourced-shopping-cart:latest" already present on machine
Normal Created 35s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Created container user-function
Normal Started 35s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Started container user-function
Normal Pulled 35s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Container image "cloudstateio/cloudstate-proxy-native-cassandra:latest" already present on machine
Normal Created 35s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Created container cloudstate-sidecar
Normal Started 35s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Started container cloudstate-sidecar
Warning Unhealthy 31s kubelet, default-235cd9d9-df1d-4cc5-b459-28892cef4871 Readiness probe failed: HTTP probe failed with statuscode: 500
=== Proxy logs ===
2021-02-02 00:04:54.921 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2021-02-02 00:04:54.944 WARN akka.actor.ActorSystemImpl - Failed to load JFR flight recorder, falling back to noop. Exception: java.lang.ClassNotFoundException: akka.remote.artery.jfr.JFRRemotingFlightRecorder
2021-02-02 00:04:54.949 INFO akka.remote.artery.tcp.ArteryTcpTransport - Remoting started with transport [Artery tcp]; listening on address [akka://[email protected]:2552] with UID [5053097932859970139]
2021-02-02 00:04:54.950 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - Starting up, Akka version [2.6.9] ...
2021-02-02 00:04:54.951 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - Started up successfully
2021-02-02 00:04:54.957 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining
2021-02-02 00:04:54.958 INFO akka.cluster.sbr.SplitBrainResolver - SBR started. Config: stableAfter: 20000 ms, strategy: KeepMajority, selfUniqueAddress: UniqueAddress(akka://[email protected]:2552,5053097932859970139), selfDc: default
2021-02-02 00:04:54.968 INFO akka.actor.ActorSystemImpl - Starting Cloudstate Proxy version [0.6.0-5-f0446998] running on [Substrate VM GraalVM 20.3.1 Java 11]
2021-02-02 00:04:54.968 INFO akka.actor.ActorSystemImpl - Registering SIGTERM handler...
2021-02-02 00:04:54.968 INFO akka.management.internal.HealthChecksImpl - Loading readiness checks [(cloudstate,io.cloudstate.proxy.HealthCheckReady)]
2021-02-02 00:04:54.968 INFO akka.management.internal.HealthChecksImpl - Loading liveness checks [(cloudstate,io.cloudstate.proxy.HealthCheckLive)]
2021-02-02 00:04:54.969 INFO akka.management.scaladsl.AkkaManagement - Binding Akka Management (HTTP) endpoint to: 172.17.0.10:8558
2021-02-02 00:04:54.969 INFO akka.management.scaladsl.AkkaManagement - Including HTTP management routes for ClusterBootstrap
2021-02-02 00:04:54.969 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Using self contact point address: http://172.17.0.10:8558
2021-02-02 00:04:54.969 INFO akka.management.scaladsl.AkkaManagement - Including HTTP management routes for HealthCheckRoutes
2021-02-02 00:04:54.975 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Initiating bootstrap procedure using kubernetes-api method...
2021-02-02 00:04:54.975 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Bootstrap using `akka.discovery` method: kubernetes-api
2021-02-02 00:04:54.975 INFO akka.management.scaladsl.AkkaManagement - Bound Akka Management (HTTP) endpoint to: 172.17.0.10:8558
2021-02-02 00:04:54.978 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Locating service members. Using discovery [akka.discovery.kubernetes.KubernetesApiServiceDiscovery], join decider [akka.management.cluster.bootstrap.LowestAddressJoinDecider], scheme [http]
2021-02-02 00:04:54.978 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Looking up [Lookup(shopping-cart-cassandra,None,Some(tcp))]
2021-02-02 00:04:54.978 INFO akka.discovery.kubernetes.KubernetesApiServiceDiscovery - Querying for pods with label selector: [cloudstate.io/stateful-service=shopping-cart-cassandra]. Namespace: [default]. Port: [None]
2021-02-02 00:04:55.025 INFO akka.actor.ActorSystemImpl - Prometheus exporter started on /0.0.0.0:9099
2021-02-02 00:04:55.052 ERROR akka.actor.OneForOneStrategy - UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at io.grpc.Status.asRuntimeException(Status.java:533)
at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426)
at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:751)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:740)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:834)
at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:8080
Caused by: java.net.ConnectException: Connection refused
at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST:Ljava_net_ConnectException_2_0002e_0003cinit_0003e_00028Ljava_lang_String_2_00029V(JNIJavaCallWrappers.java:0)
at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:834)
at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
2021-02-02 00:04:55.073 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Located service members based on: [Lookup(shopping-cart-cassandra,None,Some(tcp))]: [ResolvedTarget(172-17-0-8.default.pod.cluster.local,None,Some(/172.17.0.8))], filtered to [172-17-0-8.default.pod.cluster.local:0]
2021-02-02 00:04:55.089 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Contact point [akka://[email protected]:2552] returned [1] seed-nodes [akka://[email protected]:2552]
2021-02-02 00:04:55.089 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Joining [akka://[email protected]:2552] to existing cluster [akka://[email protected]:2552]
2021-02-02 00:04:55.089 INFO akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message [akka.management.cluster.bootstrap.internal.BootstrapCoordinator$Protocol$BootstrappingCompleted$] from Actor[akka://cloudstate-proxy/system/bootstrapCoordinator#1156786877] to Actor[akka://cloudstate-proxy/deadLetters] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cloudstate-proxy/deadLetters] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2021-02-02 00:04:55.249 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - Received InitJoinAck message from [Actor[akka://[email protected]:2552/system/cluster/core/daemon#-754755589]] to [akka://[email protected]:2552]
2021-02-02 00:04:55.375 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - Welcome from [akka://[email protected]:2552]
2021-02-02 00:04:55.978 INFO akka.cluster.sbr.SplitBrainResolver - This node is now the leader responsible for taking SBR decisions among the reachable nodes (more leaders may exist).
2021-02-02 00:04:56.257 ERROR akka.actor.OneForOneStrategy - UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at io.grpc.Status.asRuntimeException(Status.java:533)
at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426)
at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:751)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:740)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:834)
at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:8080
Caused by: java.net.ConnectException: Connection refused
at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST:Ljava_net_ConnectException_2_0002e_0003cinit_0003e_00028Ljava_lang_String_2_00029V(JNIJavaCallWrappers.java:0)
at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:834)
at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
2021-02-02 00:04:56.281 INFO akka.actor.EmptyLocalActorRef - Message [akka.cluster.ddata.Replicator$Internal$Status] from Actor[akka://[email protected]:2552/system/sharding/replicator#-1039544820] to Actor[akka://cloudstate-proxy/system/sharding/replicator] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://cloudstate-proxy/system/sharding/replicator] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2021-02-02 00:04:56.995 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - is the new leader among reachable nodes (more leaders may exist)
2021-02-02 00:04:58.300 INFO akka.actor.EmptyLocalActorRef - Message [akka.cluster.ddata.Replicator$Internal$Status] from Actor[akka://[email protected]:2552/system/sharding/replicator#-1039544820] to Actor[akka://cloudstate-proxy/system/sharding/replicator] was not delivered. [3] dead letters encountered. If this is not an expected behavior then Actor[akka://cloudstate-proxy/system/sharding/replicator] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2021-02-02 00:04:58.391 DEBUG io.cloudstate.proxy.HealthCheckReady - Error performing server manager readiness check
akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka://cloudstate-proxy/), Path(/user/server-manager-supervisor/server-manager)]
at akka.actor.ActorSelection.$anonfun$resolveOne$1(ActorSelection.scala:74)
at scala.concurrent.impl.Promise$Transformation.run(Promise.scala:447)
at scala.concurrent.ExecutionContext$parasitic$.execute(ExecutionContext.scala:164)
at scala.concurrent.impl.Promise$Transformation.submitWithValue(Promise.scala:392)
at scala.concurrent.impl.Promise$DefaultPromise.submitWithValue(Promise.scala:302)
at scala.concurrent.impl.Promise$DefaultPromise.dispatchOrAddCallbacks(Promise.scala:276)
at scala.concurrent.impl.Promise$DefaultPromise.onComplete(Promise.scala:180)
at akka.actor.ActorSelection.resolveOne(ActorSelection.scala:72)
at io.cloudstate.proxy.HealthCheckReady.check(CloudStateProxyMain.scala:48)
at io.cloudstate.proxy.HealthCheckReady.apply(CloudStateProxyMain.scala:61)
at io.cloudstate.proxy.HealthCheckReady.apply(CloudStateProxyMain.scala:38)
at scala.util.Try$.apply(Try.scala:210)
at akka.management.internal.HealthChecksImpl.runCheck(HealthChecksImpl.scala:177)
at akka.management.internal.HealthChecksImpl.$anonfun$check$2(HealthChecksImpl.scala:195)
at scala.collection.immutable.List.map(List.scala:246)
at scala.collection.immutable.List.map(List.scala:79)
at akka.management.internal.HealthChecksImpl.check(HealthChecksImpl.scala:185)
at akka.management.internal.HealthChecksImpl.readyResult(HealthChecksImpl.scala:149)
at akka.management.HealthCheckRoutes.$anonfun$routes$3(HealthCheckRoutes.scala:46)
at akka.http.scaladsl.server.directives.FutureDirectives.$anonfun$onComplete$2(FutureDirectives.scala:37)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResult$2(BasicDirectives.scala:68)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:161)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRequestContext$2(BasicDirectives.scala:45)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:161)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$2(RouteConcatenation.scala:47)
at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
at akka.http.scaladsl.util.FastFuture$.transformWith$extension(FastFuture.scala:45)
at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$2(RouteConcatenation.scala:47)
at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:41)
at akka.http.scaladsl.util.FastFuture$.transformWith$extension(FastFuture.scala:45)
at akka.http.scaladsl.util.FastFuture$.flatMap$extension(FastFuture.scala:26)
at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$1(RouteConcatenation.scala:44)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRequestContext$2(BasicDirectives.scala:45)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:161)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResultWith$2(BasicDirectives.scala:74)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:161)
at akka.http.scaladsl.server.directives.ExecutionDirectives.$anonfun$handleExceptions$2(ExecutionDirectives.scala:32)
at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:161)
at akka.http.scaladsl.server.Route$.$anonfun$asyncHandler$1(Route.scala:86)
at akka.stream.impl.fusing.MapAsyncUnordered$$anon$31.onPush(Ops.scala:1400)
at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:541)
at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:423)
at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:625)
at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:502)
at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:600)
at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:769)
at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:784)
at akka.actor.Actor.aroundReceive(Actor.scala:537)
at akka.actor.Actor.aroundReceive$(Actor.scala:535)
at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:691)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:577)
at akka.actor.ActorCell.invoke(ActorCell.scala:547)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
at akka.dispatch.Mailbox.run(Mailbox.scala:231)
at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
2021-02-02 00:04:58.392 INFO akka.management.internal.HealthChecksImpl - Check [io.cloudstate.proxy.HealthCheckReady] not ok
2021-02-02 00:04:58.676 INFO akka.actor.EmptyLocalActorRef - Message [akka.cluster.ddata.Replicator$Internal$Write] from Actor[akka://[email protected]:2552/system/sharding/replicator/$c#-1166386490] to Actor[akka://cloudstate-proxy/system/sharding/replicator] was not delivered. [4] dead letters encountered. If this is not an expected behavior then Actor[akka://cloudstate-proxy/system/sharding/replicator] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2021-02-02 00:04:58.981 INFO io.cloudstate.proxy.EntityDiscoveryManager - Received EntitySpec from user function with info: ServiceInfo(,,OpenJDK Runtime Environment 11.0.10+9,cloudstate-java-support,0.6.0-4-f0446998,0,2,UnknownFieldSet(Map()))
2021-02-02 00:04:58.981 DEBUG io.cloudstate.proxy.eventsourced.EventSourcedSupportFactory - Starting EventSourcedEntity for eventsourced-shopping-cart
2021-02-02 00:04:58.982 INFO EventingManager - Eventing support turned off in configuration
2021-02-02 00:04:58.982 INFO akka.cluster.singleton.ClusterSingletonManager - ClusterSingletonManager state change [Start -> Younger]
2021-02-02 00:04:58.982 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Post(/cart/{user_id}/items/add),*,,Vector(),UnknownFieldSet(Map()))]
2021-02-02 00:04:58.983 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Post(/cart/{user_id}/items/{product_id}/remove),,,Vector(),UnknownFieldSet(Map()))]
2021-02-02 00:04:58.983 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Get(/carts/{user_id}),,,Vector(HttpRule(,Get(/carts/{user_id}/items),,items,Vector(),UnknownFieldSet(Map()))),UnknownFieldSet(Map()))]
2021-02-02 00:04:58.984 DEBUG io.cloudstate.proxy.EntityDiscoveryManager - Starting gRPC proxy
2021-02-02 00:04:58.987 INFO io.cloudstate.proxy.EntityDiscoveryManager - CloudState proxy online at /0.0.0.0:8013
2021-02-02 00:04:46.680 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2021-02-02 00:04:46.691 WARN akka.actor.ActorSystemImpl - Failed to load JFR flight recorder, falling back to noop. Exception: java.lang.ClassNotFoundException: akka.remote.artery.jfr.JFRRemotingFlightRecorder
2021-02-02 00:04:46.697 INFO akka.remote.artery.tcp.ArteryTcpTransport - Remoting started with transport [Artery tcp]; listening on address [akka://[email protected]:2552] with UID [1154167314093528637]
2021-02-02 00:04:46.697 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - Starting up, Akka version [2.6.9] ...
2021-02-02 00:04:46.698 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - Started up successfully
2021-02-02 00:04:46.699 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining
2021-02-02 00:04:46.699 INFO akka.cluster.sbr.SplitBrainResolver - SBR started. Config: stableAfter: 20000 ms, strategy: KeepMajority, selfUniqueAddress: UniqueAddress(akka://[email protected]:2552,1154167314093528637), selfDc: default
2021-02-02 00:04:46.716 INFO akka.actor.ActorSystemImpl - Starting Cloudstate Proxy version [0.6.0-5-f0446998] running on [Substrate VM GraalVM 20.3.1 Java 11]
2021-02-02 00:04:46.716 INFO akka.actor.ActorSystemImpl - Registering SIGTERM handler...
2021-02-02 00:04:46.716 INFO akka.management.internal.HealthChecksImpl - Loading readiness checks [(cloudstate,io.cloudstate.proxy.HealthCheckReady)]
2021-02-02 00:04:46.716 INFO akka.management.internal.HealthChecksImpl - Loading liveness checks [(cloudstate,io.cloudstate.proxy.HealthCheckLive)]
2021-02-02 00:04:46.719 INFO akka.management.scaladsl.AkkaManagement - Binding Akka Management (HTTP) endpoint to: 172.17.0.8:8558
2021-02-02 00:04:46.719 INFO akka.management.scaladsl.AkkaManagement - Including HTTP management routes for ClusterBootstrap
2021-02-02 00:04:46.719 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Using self contact point address: http://172.17.0.8:8558
2021-02-02 00:04:46.719 INFO akka.management.scaladsl.AkkaManagement - Including HTTP management routes for HealthCheckRoutes
2021-02-02 00:04:46.724 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Initiating bootstrap procedure using kubernetes-api method...
2021-02-02 00:04:46.724 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Bootstrap using `akka.discovery` method: kubernetes-api
2021-02-02 00:04:46.724 INFO akka.management.scaladsl.AkkaManagement - Bound Akka Management (HTTP) endpoint to: 172.17.0.8:8558
2021-02-02 00:04:46.729 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Locating service members. Using discovery [akka.discovery.kubernetes.KubernetesApiServiceDiscovery], join decider [akka.management.cluster.bootstrap.LowestAddressJoinDecider], scheme [http]
2021-02-02 00:04:46.729 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Looking up [Lookup(shopping-cart-cassandra,None,Some(tcp))]
2021-02-02 00:04:46.729 INFO akka.discovery.kubernetes.KubernetesApiServiceDiscovery - Querying for pods with label selector: [cloudstate.io/stateful-service=shopping-cart-cassandra]. Namespace: [default]. Port: [None]
2021-02-02 00:04:46.731 INFO akka.actor.ActorSystemImpl - Prometheus exporter started on /0.0.0.0:9099
2021-02-02 00:04:46.765 ERROR akka.actor.OneForOneStrategy - UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at io.grpc.Status.asRuntimeException(Status.java:533)
at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426)
at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:751)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:740)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:834)
at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:8080
Caused by: java.net.ConnectException: Connection refused
at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST:Ljava_net_ConnectException_2_0002e_0003cinit_0003e_00028Ljava_lang_String_2_00029V(JNIJavaCallWrappers.java:0)
at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:834)
at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
2021-02-02 00:04:46.786 INFO akka.discovery.kubernetes.KubernetesApiServiceDiscovery - No targets found from pod list. Is the correct port name configured? Current configuration: [None]. Ports on pods: [Set(ContainerPort(Some(user-port),8080), ContainerPort(Some(grpc-http-proxy),8013), ContainerPort(Some(cs-metrics),9099))]
2021-02-02 00:04:46.786 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Located service members based on: [Lookup(shopping-cart-cassandra,None,Some(tcp))]: [], filtered to []
2021-02-02 00:04:47.740 INFO akka.management.cluster.bootstrap.LowestAddressJoinDecider - Discovered [0] contact points, confirmed [0], which is less than the required [1], retrying
2021-02-02 00:04:47.811 ERROR akka.actor.OneForOneStrategy - UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at io.grpc.Status.asRuntimeException(Status.java:533)
at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426)
at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:751)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:740)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:834)
at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:8080
Caused by: java.net.ConnectException: Connection refused
at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST:Ljava_net_ConnectException_2_0002e_0003cinit_0003e_00028Ljava_lang_String_2_00029V(JNIJavaCallWrappers.java:0)
at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:834)
at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
2021-02-02 00:04:47.930 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Looking up [Lookup(shopping-cart-cassandra,None,Some(tcp))]
2021-02-02 00:04:47.930 INFO akka.discovery.kubernetes.KubernetesApiServiceDiscovery - Querying for pods with label selector: [cloudstate.io/stateful-service=shopping-cart-cassandra]. Namespace: [default]. Port: [None]
2021-02-02 00:04:47.934 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Located service members based on: [Lookup(shopping-cart-cassandra,None,Some(tcp))]: [ResolvedTarget(172-17-0-8.default.pod.cluster.local,None,Some(/172.17.0.8))], filtered to [172-17-0-8.default.pod.cluster.local:0]
2021-02-02 00:04:47.955 INFO akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes - Bootstrap request from 172.17.0.8:48894: Contact Point returning 0 seed-nodes []
2021-02-02 00:04:47.956 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Contact point [akka://[email protected]:2552] returned [0] seed-nodes []
2021-02-02 00:04:48.980 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Looking up [Lookup(shopping-cart-cassandra,None,Some(tcp))]
2021-02-02 00:04:48.981 INFO akka.discovery.kubernetes.KubernetesApiServiceDiscovery - Querying for pods with label selector: [cloudstate.io/stateful-service=shopping-cart-cassandra]. Namespace: [default]. Port: [None]
2021-02-02 00:04:48.992 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Located service members based on: [Lookup(shopping-cart-cassandra,None,Some(tcp))]: [ResolvedTarget(172-17-0-8.default.pod.cluster.local,None,Some(/172.17.0.8))], filtered to [172-17-0-8.default.pod.cluster.local:0]
2021-02-02 00:04:49.080 INFO akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes - Bootstrap request from 172.17.0.8:48894: Contact Point returning 0 seed-nodes []
2021-02-02 00:04:49.080 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Contact point [akka://[email protected]:2552] returned [0] seed-nodes []
2021-02-02 00:04:49.739 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Initiating new cluster, self-joining [akka://[email protected]:2552]. Other nodes are expected to locate this cluster via continued contact-point probing.
2021-02-02 00:04:49.740 INFO akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message [akka.management.cluster.bootstrap.internal.BootstrapCoordinator$Protocol$BootstrappingCompleted$] from Actor[akka://cloudstate-proxy/system/bootstrapCoordinator#1480424403] to Actor[akka://cloudstate-proxy/deadLetters] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cloudstate-proxy/deadLetters] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2021-02-02 00:04:49.740 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - Node [akka://[email protected]:2552] is JOINING itself (with roles [dc-default]) and forming new cluster
2021-02-02 00:04:49.740 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - is the new leader among reachable nodes (more leaders may exist)
2021-02-02 00:04:49.740 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - Leader is moving node [akka://[email protected]:2552] to [Up]
2021-02-02 00:04:49.740 INFO akka.cluster.sbr.SplitBrainResolver - This node is now the leader responsible for taking SBR decisions among the reachable nodes (more leaders may exist).
2021-02-02 00:04:50.235 INFO io.cloudstate.proxy.EntityDiscoveryManager - Received EntitySpec from user function with info: ServiceInfo(,,OpenJDK Runtime Environment 11.0.10+9,cloudstate-java-support,0.6.0-4-f0446998,0,2,UnknownFieldSet(Map()))
2021-02-02 00:04:50.236 DEBUG io.cloudstate.proxy.eventsourced.EventSourcedSupportFactory - Starting EventSourcedEntity for eventsourced-shopping-cart
2021-02-02 00:04:50.237 INFO EventingManager - Eventing support turned off in configuration
2021-02-02 00:04:50.238 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Post(/cart/{user_id}/items/add),*,,Vector(),UnknownFieldSet(Map()))]
2021-02-02 00:04:50.238 INFO akka.cluster.singleton.ClusterSingletonManager - Singleton manager starting singleton actor [akka://cloudstate-proxy/system/sharding/eventsourced-shopping-cartCoordinator/singleton]
2021-02-02 00:04:50.238 INFO akka.cluster.singleton.ClusterSingletonManager - ClusterSingletonManager state change [Start -> Oldest]
2021-02-02 00:04:50.238 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Post(/cart/{user_id}/items/{product_id}/remove),,,Vector(),UnknownFieldSet(Map()))]
2021-02-02 00:04:50.238 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Get(/carts/{user_id}),,,Vector(HttpRule(,Get(/carts/{user_id}/items),,items,Vector(),UnknownFieldSet(Map()))),UnknownFieldSet(Map()))]
2021-02-02 00:04:50.238 INFO akka.cluster.sharding.DDataShardCoordinator - ShardCoordinator was moved to the active state State(Map())
2021-02-02 00:04:50.239 DEBUG io.cloudstate.proxy.EntityDiscoveryManager - Starting gRPC proxy
2021-02-02 00:04:50.241 INFO io.cloudstate.proxy.EntityDiscoveryManager - CloudState proxy online at /0.0.0.0:8013
2021-02-02 00:04:55.086 INFO akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes - Bootstrap request from 172.17.0.9:50312: Contact Point returning 1 seed-nodes [akka://[email protected]:2552]
2021-02-02 00:04:55.088 INFO akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes - Bootstrap request from 172.17.0.10:58948: Contact Point returning 1 seed-nodes [akka://[email protected]:2552]
2021-02-02 00:04:55.125 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - Received InitJoin message from [Actor[akka://[email protected]:2552/system/cluster/core/daemon/joinSeedNodeProcess-1#188025117]] to [akka://[email protected]:2552]
2021-02-02 00:04:55.125 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - Sending InitJoinAck message from node [akka://[email protected]:2552] to [Actor[akka://[email protected]:2552/system/cluster/core/daemon/joinSeedNodeProcess-1#188025117]] (version [2.6.9])
2021-02-02 00:04:55.249 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - Received InitJoin message from [Actor[akka://[email protected]:2552/system/cluster/core/daemon/joinSeedNodeProcess-1#933739104]] to [akka://[email protected]:2552]
2021-02-02 00:04:55.249 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - Sending InitJoinAck message from node [akka://[email protected]:2552] to [Actor[akka://[email protected]:2552/system/cluster/core/daemon/joinSeedNodeProcess-1#933739104]] (version [2.6.9])
2021-02-02 00:04:55.374 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - Node [akka://[email protected]:2552] is JOINING, roles [dc-default]
2021-02-02 00:04:55.379 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - Node [akka://[email protected]:2552] is JOINING, roles [dc-default]
2021-02-02 00:04:55.859 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - Leader is moving node [akka://[email protected]:2552] to [Up]
2021-02-02 00:04:55.860 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - Leader is moving node [akka://[email protected]:2552] to [Up]
2021-02-02 00:04:55.860 INFO akka.cluster.sbr.SplitBrainResolver - This node is not the leader any more and not responsible for taking SBR decisions.
2021-02-02 00:04:56.880 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - is no longer leader
2021-02-02 00:05:03.690 ERROR akka.cluster.sharding.DDataShardCoordinator - The ShardCoordinator was unable to update a distributed state within 'updating-state-timeout': 5000 millis (retrying). Perhaps the ShardRegion has not started on all active nodes yet? event=ShardRegionRegistered(Actor[akka://[email protected]:2552/system/sharding/eventsourced-shopping-cart#1207442416])
2021-02-02 00:05:09.125 WARN akka.actor.ActorSystemImpl - Failed to load JFR flight recorder, falling back to noop. Exception: java.lang.ClassNotFoundException: akka.cluster.sharding.internal.jfr.JFRShardingFlightRecorder
2021-02-02 00:05:09.130 INFO com.datastax.driver.core.GuavaCompatibility - Detected Guava >= 19 in the classpath, using modern compatibility layer
2021-02-02 00:05:09.139 INFO com.datastax.driver.core - DataStax Java driver 3.7.2 for Apache Cassandra
2021-02-02 00:05:09.140 INFO com.datastax.driver.core.Native - Could not load JNR C Library, native system calls through this library will not be available (set this logger level to DEBUG to see the full stack trace).
2021-02-02 00:05:09.140 INFO com.datastax.driver.core.ClockFactory - Using java.lang.System clock to generate timestamps.
2021-02-02 00:05:09.140 INFO com.datastax.driver.core.ClockFactory - Using java.lang.System clock to generate timestamps.
2021-02-02 00:05:09.142 INFO com.datastax.driver.core.NettyUtil - Did not find Netty's native epoll transport in the classpath, defaulting to NIO.
2021-02-02 00:05:09.143 INFO com.datastax.driver.core.Native - Could not load JNR POSIX Library, native system calls through this library will not be available (set this logger level to DEBUG to see the full stack trace).
2021-02-02 00:05:09.144 INFO com.datastax.driver.core.utils.UUIDs - PID obtained through JMX: 1
2021-02-02 00:05:09.144 INFO com.datastax.driver.core.ClockFactory - Using java.lang.System clock to generate timestamps.
2021-02-02 00:05:09.369 INFO com.datastax.driver.core.policies.DCAwareRoundRobinPolicy - Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
2021-02-02 00:05:09.369 INFO com.datastax.driver.core.Cluster - New Cassandra host cassandra.default.svc.cluster.local/10.107.191.162:9042 added
2021-02-02 00:05:09.409 INFO com.datastax.driver.core.policies.DCAwareRoundRobinPolicy - Using data-center name 'datacenter1' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
2021-02-02 00:05:09.409 INFO com.datastax.driver.core.Cluster - New Cassandra host cassandra.default.svc.cluster.local/10.107.191.162:9042 added
2021-02-02 00:05:22.470 INFO com.datastax.driver.core.ClockFactory - Using java.lang.System clock to generate timestamps.
2021-02-02 00:05:22.590 INFO com.datastax.driver.core.ClockFactory - Using java.lang.System clock to generate timestamps.
2021-02-02 00:05:29.149 ERROR io.cloudstate.proxy.eventsourced.EventSourcedEntity - Persistence failure when replaying events for persistenceId [eventsourced-shopping-cart|test1]. Last known sequence number [0]
akka.pattern.CircuitBreaker$$anon$13: Circuit Breaker Timed out.
2021-02-02 00:05:29.150 INFO akka.actor.LocalActorRef - Message [io.cloudstate.proxy.entity.EntityCommand] from Actor[akka://[email protected]:2552/temp/eventsourced-shopping-cart$a] to Actor[akka://cloudstate-proxy/system/sharding/eventsourced-shopping-cart/87/test1/entity#-425630766] was not delivered. [2] dead letters encountered. If this is not an expected behavior then Actor[akka://cloudstate-proxy/system/sharding/eventsourced-shopping-cart/87/test1/entity#-425630766] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2021-02-02 00:04:54.928 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2021-02-02 00:04:54.932 WARN akka.actor.ActorSystemImpl - Failed to load JFR flight recorder, falling back to noop. Exception: java.lang.ClassNotFoundException: akka.remote.artery.jfr.JFRRemotingFlightRecorder
2021-02-02 00:04:54.949 INFO akka.remote.artery.tcp.ArteryTcpTransport - Remoting started with transport [Artery tcp]; listening on address [akka://[email protected]:2552] with UID [-2365786287080537531]
2021-02-02 00:04:54.949 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - Starting up, Akka version [2.6.9] ...
2021-02-02 00:04:54.950 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - Started up successfully
2021-02-02 00:04:54.950 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining
2021-02-02 00:04:54.951 INFO akka.cluster.sbr.SplitBrainResolver - SBR started. Config: stableAfter: 20000 ms, strategy: KeepMajority, selfUniqueAddress: UniqueAddress(akka://[email protected]:2552,-2365786287080537531), selfDc: default
2021-02-02 00:04:54.966 INFO akka.actor.ActorSystemImpl - Starting Cloudstate Proxy version [0.6.0-5-f0446998] running on [Substrate VM GraalVM 20.3.1 Java 11]
2021-02-02 00:04:54.967 INFO akka.actor.ActorSystemImpl - Registering SIGTERM handler...
2021-02-02 00:04:54.967 INFO akka.management.internal.HealthChecksImpl - Loading readiness checks [(cloudstate,io.cloudstate.proxy.HealthCheckReady)]
2021-02-02 00:04:54.967 INFO akka.management.internal.HealthChecksImpl - Loading liveness checks [(cloudstate,io.cloudstate.proxy.HealthCheckLive)]
2021-02-02 00:04:54.969 INFO akka.management.scaladsl.AkkaManagement - Binding Akka Management (HTTP) endpoint to: 172.17.0.9:8558
2021-02-02 00:04:54.969 INFO akka.management.scaladsl.AkkaManagement - Including HTTP management routes for ClusterBootstrap
2021-02-02 00:04:54.969 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Using self contact point address: http://172.17.0.9:8558
2021-02-02 00:04:54.969 INFO akka.management.scaladsl.AkkaManagement - Including HTTP management routes for HealthCheckRoutes
2021-02-02 00:04:54.975 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Initiating bootstrap procedure using kubernetes-api method...
2021-02-02 00:04:54.975 INFO akka.management.cluster.bootstrap.ClusterBootstrap - Bootstrap using `akka.discovery` method: kubernetes-api
2021-02-02 00:04:54.975 INFO akka.management.scaladsl.AkkaManagement - Bound Akka Management (HTTP) endpoint to: 172.17.0.9:8558
2021-02-02 00:04:54.978 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Locating service members. Using discovery [akka.discovery.kubernetes.KubernetesApiServiceDiscovery], join decider [akka.management.cluster.bootstrap.LowestAddressJoinDecider], scheme [http]
2021-02-02 00:04:54.979 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Looking up [Lookup(shopping-cart-cassandra,None,Some(tcp))]
2021-02-02 00:04:54.979 INFO akka.discovery.kubernetes.KubernetesApiServiceDiscovery - Querying for pods with label selector: [cloudstate.io/stateful-service=shopping-cart-cassandra]. Namespace: [default]. Port: [None]
2021-02-02 00:04:55.001 INFO akka.actor.ActorSystemImpl - Prometheus exporter started on /0.0.0.0:9099
2021-02-02 00:04:55.065 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Located service members based on: [Lookup(shopping-cart-cassandra,None,Some(tcp))]: [ResolvedTarget(172-17-0-8.default.pod.cluster.local,None,Some(/172.17.0.8))], filtered to [172-17-0-8.default.pod.cluster.local:0]
2021-02-02 00:04:55.066 ERROR akka.actor.OneForOneStrategy - UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at io.grpc.Status.asRuntimeException(Status.java:533)
at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426)
at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:751)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:740)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:834)
at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:8080
Caused by: java.net.ConnectException: Connection refused
at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST:Ljava_net_ConnectException_2_0002e_0003cinit_0003e_00028Ljava_lang_String_2_00029V(JNIJavaCallWrappers.java:0)
at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:834)
at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
2021-02-02 00:04:55.086 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Contact point [akka://[email protected]:2552] returned [1] seed-nodes [akka://[email protected]:2552]
2021-02-02 00:04:55.087 INFO akka.management.cluster.bootstrap.internal.BootstrapCoordinator - Joining [akka://[email protected]:2552] to existing cluster [akka://[email protected]:2552]
2021-02-02 00:04:55.087 INFO akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef - Message [akka.management.cluster.bootstrap.internal.BootstrapCoordinator$Protocol$BootstrappingCompleted$] from Actor[akka://cloudstate-proxy/system/bootstrapCoordinator#15500066] to Actor[akka://cloudstate-proxy/deadLetters] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://cloudstate-proxy/deadLetters] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
2021-02-02 00:04:55.378 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - Received InitJoinAck message from [Actor[akka://[email protected]:2552/system/cluster/core/daemon#-754755589]] to [akka://[email protected]:2552]
2021-02-02 00:04:55.380 INFO akka.cluster.Cluster - Cluster Node [akka://[email protected]:2552] - Welcome from [akka://[email protected]:2552]
2021-02-02 00:04:56.179 ERROR akka.actor.OneForOneStrategy - UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at io.grpc.Status.asRuntimeException(Status.java:533)
at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426)
at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:751)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:740)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:834)
at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:8080
Caused by: java.net.ConnectException: Connection refused
at com.oracle.svm.jni.JNIJavaCallWrappers.jniInvoke_VA_LIST:Ljava_net_ConnectException_2_0002e_0003cinit_0003e_00028Ljava_lang_String_2_00029V(JNIJavaCallWrappers.java:0)
at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:779)
at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
at io.grpc.netty.shaded.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:834)
at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:519)
at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:192)
2021-02-02 00:04:58.583 INFO akka.management.internal.HealthChecksImpl - Check [io.cloudstate.proxy.HealthCheckReady] not ok
2021-02-02 00:04:58.672 INFO io.cloudstate.proxy.EntityDiscoveryManager - Received EntitySpec from user function with info: ServiceInfo(,,OpenJDK Runtime Environment 11.0.10+9,cloudstate-java-support,0.6.0-4-f0446998,0,2,UnknownFieldSet(Map()))
2021-02-02 00:04:58.672 DEBUG io.cloudstate.proxy.eventsourced.EventSourcedSupportFactory - Starting EventSourcedEntity for eventsourced-shopping-cart
2021-02-02 00:04:58.674 INFO akka.cluster.singleton.ClusterSingletonManager - ClusterSingletonManager state change [Start -> Younger]
2021-02-02 00:04:58.674 INFO EventingManager - Eventing support turned off in configuration
2021-02-02 00:04:58.674 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Post(/cart/{user_id}/items/add),*,,Vector(),UnknownFieldSet(Map()))]
2021-02-02 00:04:58.675 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Post(/cart/{user_id}/items/{product_id}/remove),,,Vector(),UnknownFieldSet(Map()))]
2021-02-02 00:04:58.675 INFO HttpApi(akka://cloudstate-proxy) - Using configured HTTP API endpoint using [HttpRule(,Get(/carts/{user_id}),,,Vector(HttpRule(,Get(/carts/{user_id}/items),,items,Vector(),UnknownFieldSet(Map()))),UnknownFieldSet(Map()))]
2021-02-02 00:04:58.676 DEBUG io.cloudstate.proxy.EntityDiscoveryManager - Starting gRPC proxy
2021-02-02 00:04:58.680 INFO io.cloudstate.proxy.EntityDiscoveryManager - CloudState proxy online at /0.0.0.0:8013
2021-02-02 00:05:09.122 DEBUG io.cloudstate.proxy.HttpApi$HttpEndpoint - Received HTTP request [/carts/test1] for [com.example.shoppingcart.ShoppingCart.GetCart]
* Request Field: [com.example.shoppingcart.GetShoppingCart.user_id] = [test1]
2021-02-02 00:05:09.122 DEBUG io.cloudstate.proxy.Serve$ - Received gRPC request [/com.example.shoppingcart.ShoppingCart/GetCart]
2021-02-02 00:05:29.134 INFO akka.actor.ActorSystemImpl - Request timeout encountered for request [GET /carts/test1 Empty]
=== User function logs ===
2021-02-02 00:04:56.069 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$AddLineItem
2021-02-02 00:04:56.075 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.google.protobuf.Empty
2021-02-02 00:04:56.077 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$RemoveLineItem
2021-02-02 00:04:56.079 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$GetShoppingCart
2021-02-02 00:04:56.081 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$Cart
[INFO] [02/02/2021 00:04:58.893] [StatefulService-akka.actor.default-dispatcher-6] [akka.actor.ActorSystemImpl(StatefulService)] Received discovery call from [cloudstate-proxy-core 0.6.0-5-f0446998] supporting Cloudstate protocol 0.2
2021-02-02 00:04:47.408 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$AddLineItem
2021-02-02 00:04:47.411 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.google.protobuf.Empty
2021-02-02 00:04:47.413 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$RemoveLineItem
2021-02-02 00:04:47.414 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$GetShoppingCart
2021-02-02 00:04:47.416 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$Cart
[INFO] [02/02/2021 00:04:50.179] [StatefulService-akka.actor.default-dispatcher-6] [akka.actor.ActorSystemImpl(StatefulService)] Received discovery call from [cloudstate-proxy-core 0.6.0-5-f0446998] supporting Cloudstate protocol 0.2
[ERROR] [02/02/2021 00:05:29.161] [StatefulService-akka.actor.default-dispatcher-10] [EventSourcedImpl] Terminating entity due to unexpected failure
io.cloudstate.javasupport.impl.EntityExceptions$EntityException: Protocol error: Expected Init message
at io.cloudstate.javasupport.impl.EntityExceptions$ProtocolException$.apply(EntityExceptions.scala:46)
at io.cloudstate.javasupport.impl.eventsourced.EventSourcedImpl.$anonfun$handle$1(EventSourcedImpl.scala:125)
at akka.stream.impl.fusing.Map$$anon$1.onPush(Ops.scala:53)
at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:541)
at akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:527)
at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:390)
at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:625)
at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:502)
at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:600)
at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:769)
at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$shortCircuitBatch(ActorGraphInterpreter.scala:759)
at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:785)
at akka.actor.Actor.aroundReceive(Actor.scala:537)
at akka.actor.Actor.aroundReceive$(Actor.scala:535)
at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:691)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:577)
at akka.actor.ActorCell.invoke(ActorCell.scala:547)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
at akka.dispatch.Mailbox.run(Mailbox.scala:231)
at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
2021-02-02 00:04:55.907 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$AddLineItem
2021-02-02 00:04:55.911 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.google.protobuf.Empty
2021-02-02 00:04:55.913 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$RemoveLineItem
2021-02-02 00:04:55.915 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$GetShoppingCart
2021-02-02 00:04:55.916 DEBUG io.cloudstate.javasupport.impl.AnySupport - Attempting to load com.google.protobuf.Message class com.example.shoppingcart.Shoppingcart$Cart
[INFO] [02/02/2021 00:04:58.609] [StatefulService-akka.actor.default-dispatcher-5] [akka.actor.ActorSystemImpl(StatefulService)] Received discovery call from [cloudstate-proxy-core 0.6.0-5-f0446998] supporting Cloudstate protocol 0.2
Exited with code exit status 1
CircleCI received exit code 1
Judging by these logs it seems to me that the proxy is not activating the entity first.
Yeah, it's not managing to initialise the entity first. I think the timeout on replaying events will be the issue:
ERROR io.cloudstate.proxy.eventsourced.EventSourcedEntity - Persistence failure when replaying events for persistenceId [eventsourced-shopping-cart|test1]. Last known sequence number [0]
akka.pattern.CircuitBreaker$$anon$13: Circuit Breaker Timed out.
The user function error, that it expected an init message, is not as useful without saying what it received instead.