fission-workflows icon indicating copy to clipboard operation
fission-workflows copied to clipboard

Published 20 hours ago • verified publisher icon fission

Nats connection closed

Open vishal-biyani opened this issue 6 years ago • 1 comments

After certain time the workflow pod looses connection to the Nats. This is fixed after the workflow pod is restarted.

time="2018-09-05T04:23:56Z" level=info msg="Event added: WORKFLOW_CREATED" aggregate=workflow/29dc36fc-b037-11e8-8f31-42010aa0001c nats.subject=workflow.29dc36fc-b037-11e8-8f31-42010aa0001c parent=/
time="2018-09-05T04:23:56Z" level=error msg="Request error: nats: connection closed"
time="2018-09-05T04:23:56Z" level=error msg="failed to specialize: failed to specialize package: failed to store workflow internally: nats: connection closed"

This leads to 502 on any workflow function being called. We should have some way to check and restore connection to nats so that the user does not have to face 502.

vishal-biyani avatar Sep 05 '18 04:09 vishal-biyani

Thanks @vishal-biyani for the issue! Most of the reconnecting logic is handled by the nats client, but apparently not in all cases. To fix this, we should indeed add functionality to reconnect to NATS upon connection loss.

erwinvaneyk avatar Sep 06 '18 10:09 erwinvaneyk