nomad
nomad copied to clipboard
hashicorp
[feature]: cli job run retry on "Error fetching deployment"
Proposal
Add an option --retry (default 3) for monitoring the job run status.
Use-cases
We run a reverse proxy as a cetralized entry point for our devs to the nomad API. The proxy itself runs on nomad.
If our CI re-deploys that proxy accessing the API through that proxy, even with canary and rolling update it might be the cli errors out with "Error fetching deployment".
We added a retry on error in our CI, but we wished the CLI job monitoring would just retry a couple of times by itself e.g. 3 times with a delay of a second.
Attempted Solutions
canary and rolling update
Hey @resmo
This would be a great feature to have so thanks for making this, we'll address this internally and get back to you with our thoughts 👍. Would you be willing to raise a PR for this?
@bubejur we'll update issues when we're working on them. This isn't currently on our immediate roadmap.
We upgraded our CI pipeline to use nomad cli 1.5.5 (from 1.4.4).
With 1.4.4 we could simply call nomad job run <jobfile>.
With 1.5.5 we always get the error "Error fetching deployment" after a couple of seconds.
I guess we have to implement a retry logic, beacuse the pipeline always fails now.
So ... this feature would be much appreciated.
Doubling down on @vkrebs-wktaa comment. I just upgraded from 1.3.x to 1.5.6 and now I see this error almost every time I nomad job run. Seems like a major regression.
Hi all 👋
Just noting here that, while investigating #17320, I noticed that the deployment monitor was not outputting the actual error that happened, which makes it hard to understand what problem is happening, so I've opened #17348 to improve this.
A retry mechanism would indeed be helpful, so I'm keeping this one opened.
