Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'

[lejoow]

I am not 100% sure if this is the issue specifics to my local settings or not. But every time I try to deploy the item-svc or bidding-svc to my Google KBE, I am getting this exception.

Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'

[TimMoore]

I think I've found the problem. GCP enables Role-Based Access Control (aka RBAC) by default in new Kubernetes clusters. This requires configuring roles to allow access to resources within the Kubernetes API. Lightbend Orchestration (specifically, the Akka Cluster Bootstrap component) uses the Kubernetes API to automatically form a cluster. Without binding your pods' service account to a role that grants access to the required resources, these API calls fail.

To correct this, you'll need to follow the instructions on this page for configuring role-based access control: https://developer.lightbend.com/docs/akka-management/current/discovery.html#discovery-method-kubernetes-api

For example, you can supply these Kubernetes resources to kubectl apply:

---
#
# Create a role, `pod-reader`, that can list pods and
# bind the default service account in the `default` namespace
# to that role.
#

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: read-pods
subjects:
# Note the `name` line below. The first default refers to the namespace. The second refers to the service account name.
# For instance, `name: system:serviceaccount:myns:default` would refer to the default service account in namespace `myns`
- kind: User
  name: system:serviceaccount:default:default
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Once you've created this role and role binding, your Lagom services should start correctly. You might want to tailor this to your specific security needs.

@lejoow please let us know if that fixes the problem for you. Even if it does, we can leave this issue open until we update the documentation.

[lejoow]

@TimMoore fantastic. Let me try this fix today and come back to you on this post. Thanks a lot!

[lejoow]

hmm... I don't get that error anymore, but I am getting the different errors now:

2018-05-10T17:54:27.363Z [[37minfo[0m] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=application-akka.actor.default-dispatcher-24, akkaTimestamp=17:54:27.363UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=application] - Exceeded stable margins without locating seed-nodes, however this node is NOT the lowest address out of the discovered IPs in this deployment, thus NOT joining self. Expecting node Some(ResolvedTarget(10.16.0.13,Some(10002))) (out of List(ResolvedTarget(10.16.0.13,Some(10002)), ResolvedTarget(10.16.1.48,Some(10002)))) to perform the self-join and initiate the cluster.

I am still investigating..

[lejoow]

It worked! I just had to re-deploy. Thanks! Btw, when I tried to create the role by applying that file you pasted, I ran into "pod-reader forbidden" error message, which I resolved by following the instructions here:

kubectl create clusterrolebinding cluster-admin-binding
--clusterrole=cluster-admin
--user=$primary_account https://github.com/jcbsmpsn/gke-rbac-walkthrough/blob/master/create-roles.md

[TimMoore]

Let's leave this open until the documentation is updated. Thanks!

[TimMoore]

Added an issue for the Akka Management documentation to note that Google Cloud users will need to grant themselves the cluster-admin ClusterRole: https://github.com/akka/akka-management/issues/199

[Canislupax]

That solved my problem! Thanx a lot Tim! You`re the best!

[idoshamun]

@TimMoore I followed your suggestion but now I receive:

14:47:46.025 [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'ports'

For what it's worth, I am trying to run Lagom with Istio

[idoshamun]

Disabling Istio resolves this issue, I will try to look further and share results

[normenmueller]

@TimMoore @lejoow I have the same issue running on Minikube. Here are my deployment steps:

$ sbt docker:publishLocal

$ minikube delete
$ minikube start --memory 8192
$ eval $(minikube docker-env)          <<< Should this command come before docker:publishLocal?
$ minikube addons enable ingress
$ kubectl apply -f RBAC.yml            <<< The file posted by @TimMoore

$ sbt "deploy minikube"

$ kubectl cluster-info
$ kubectl get deployments
$ kubectl get pods
$ echo "http://$(minikube ip)"

All Pods have status Running.

Then I had a look at the minikube dashboard and all my services are marked with

Readiness probe failed: Get http://172.17.0.8:10002/platform-tooling/ready: dial tcp 172.17.0.8:10002: getsockopt: connection refused

Then I had a look in the logs at the minikube dashboard and get:

[error] akka.actor.ActorSystemImpl [sourceThread=application-akka.actor.default-dispatcher-2, akkaTimestamp=16:32:22.971UTC, akkaSource=akka.actor.ActorSystemImpl(application), sourceActorSystem=application] - Failed to unmarshal Kubernetes API response status [400 Bad Request]; check RBAC settings

[Bunyod]

@TimMoore @lejoow @normenmueller I'm also facing with the same getsockopt: connection refused issue for Minikube. I follow this link.

Does anybody have an idea how to fix it?

[OguzhanSaltik]

We have the same error, due to Lightbend Orchestration documentation.

[TimMoore]

The getsockopt: connection refused message itself only indicates that the service is not running. Of course, there could be many reasons for that. You'll need to look at the logs for the pod to understand more. I'd like to keep this issue thread focused on the original issue reported, which has a solution described in a comment above: https://github.com/lagom/online-auction-scala/issues/78#issuecomment-387933707

If you're having another problem, the best place to start is by posting an issue on the Discourse forum: https://discuss.lightbend.com/

[bahtiyarerden]

We are trying to deploy our lagom application with kubernetes. But we could not fix this issue. We clone this repository (chirper) and deploy with minikube to be sure the problem does not about our configuration. It is the result:

And this is the log file:

2018-06-21T12:36:25.703Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-21, akkaTimestamp=12:36:25.703UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:26.756Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-15, akkaTimestamp=12:36:26.756UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:26.883Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-21, akkaTimestamp=12:36:26.883UTC, akkaSource=akka.tcp://[email protected]:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:27.802Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-18, akkaTimestamp=12:36:27.801UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:28.839Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:28.839UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:28.882Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-14, akkaTimestamp=12:36:28.882UTC, akkaSource=akka.tcp://[email protected]:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:29.863Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-14, akkaTimestamp=12:36:29.863UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:30.883Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-4, akkaTimestamp=12:36:30.882UTC, akkaSource=akka.tcp://[email protected]:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:30.936Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-4, akkaTimestamp=12:36:30.935UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:32.051Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-18, akkaTimestamp=12:36:32.051UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:32.893Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:32.892UTC, akkaSource=akka.tcp://[email protected]:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:33.089Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:33.088UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:34.116Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-18, akkaTimestamp=12:36:34.115UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:34.873Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:34.872UTC, akkaSource=akka.tcp://[email protected]:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:35.159Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:35.157UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:36.257Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-15, akkaTimestamp=12:36:36.257UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:36.873Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:36.873UTC, akkaSource=akka.tcp://[email protected]:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:37.304Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:37.301UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:38.332Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-18, akkaTimestamp=12:36:38.332UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:38.885Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-2, akkaTimestamp=12:36:38.885UTC, akkaSource=akka.tcp://[email protected]:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:39.378Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-18, akkaTimestamp=12:36:39.376UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:40.420Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-18, akkaTimestamp=12:36:40.420UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:40.882Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-15, akkaTimestamp=12:36:40.882UTC, akkaSource=akka.tcp://[email protected]:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:41.468Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-21, akkaTimestamp=12:36:41.468UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'

I think the problem is about configuration that given below. My microservices that do not have this configuration are working without any error. When i look the logs then i see the akka management port is 10002. The readiness prob also check this port and getting getsockopt: connection refused error.

The application.conf file configuration:

akka.actor.serialization-bindings {
  "akka.Done"                 = akka-misc
  "akka.actor.Address"        = akka-misc
  "akka.remote.UniqueAddress" = akka-misc
}

[ale222]

Trying to deploy online-auction, getting:

akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=application-akka.actor.default-dispatcher-5, akkaTimestamp=12:27:31.874UTC, akkaSource=akka.tcp://[email protected]:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=application] - Discovered [1] observation, which is less than the required [2], retrying (interval: 1.000 s)

any ideas?

[zhangzhonglai]

same error because of akka/akka-management#222 when using sbt-reactive-app >= 1.2.0 if someone encounter the error like me, simply back to use sbt-reactive-app 1.1.0 will temporary fix the error now.

[ktoso]

The related issue #222 was resolved, 0.15 should fix it then. Thanks for digging into this!

[OguzhanSaltik]

Great news @ktoso , when will be 0.15 version applied to the Lagom?

[ktoso]

I'll have to rely this to @TimMoore, not sure on release schedule of intermediate deps etc.

[TimMoore]

Lagom has no dependency on akka-management. That comes in by way of Lightbend Orchestration, so I think @mitkus is the right one to ask.

[OguzhanSaltik]

You are right, the related fix is part of the sbt-reactive-app plugin. We look forward to the version updates, thanks to everyone.