constructr-consul icon indicating copy to clipboard operation
constructr-consul copied to clipboard

Session is NOT explicitly destroyed during clean shutdown

Open pocman opened this issue 7 years ago • 3 comments

Hi,

  • I'm using constructr inside nomad Tasks running Lagom inside a docker image.
  • I might have multiple Tasks on one node.
  • I have one consul agent per node.
  • I'm currently not using the agent-name configuration but my constructr.coordination.{host/port} points to the local consul agent, So I hope it's added automatically.

My issue is that when running nomad stop, the sessions associated with each tasks are not destroyed and I must wait for the ttl to expires.

In consul documentation, it's indicated that a session will be invalidated:

  • Node is deregistered
  • Any of the health checks are deregistered
  • Any of the health checks go to the critical state
  • Session is explicitly destroyed
  • TTL expires, if applicable

Node deregistering doesn't solve my issue since the node is still running after the nomad stop, we can't add healthcheck in the conf and session is not destroyed on clean shutdown.

pocman avatar Nov 08 '17 10:11 pocman

Hi, thanks for your issue. You are right, we should destroy session on system shutdown. I'm going to open an issue at https://github.com/hseeberger/constructr, suggesting the addition of a close() method to Coordination. This would allow us to release Consul-related resources in an elegant way by relying on the postStop() handler of ConstructrMachine. Also, I'm going to close #50 because of the above reasoning. Besides that, it seems that we should use a different HTTP client to communicate with Consul, due to akka-http provoking the java.lang.IllegalStateException: cannot create children while terminating or terminated exception.

franbh avatar Nov 14 '17 08:11 franbh

Why not use registerOnTermination instead of postStop ?

pocman avatar Nov 14 '17 08:11 pocman

That would be a quick solution (provided that we switch to a different HTTP client), but I think that Constructr core would benefit of having a clear way of releasing such resources.

franbh avatar Nov 14 '17 08:11 franbh