habitat
habitat copied to clipboard
Feature Request: Log to stdout when supervisor starts successfully
Right now, hab-sup logs "Starting http-gateway on..." as the last line as it's starting up. Can we get a log message to the effect of "supervisor started successfully"?
It's in support of this pattern:
# that long, period, but just in case.
timeout 300s tail -f /var/log/messages | grep -m1 “Starting http-gateway”
# Give it one additional second, as we don’t have a post-started message.
sleep 1
/bin/hab svc load ${base_pkg} --channel stable --strategy at-once --force```
(Trying to deal with a race condition)
Other (potentially better, if less portable) solutions would be supporting SystemD's notify
or dbus
service types. At that point, I can simply ask SystemD to block me until the supervisor service comes up.
Definitely agree that integration with systemd is good, but as an alternative to the above pattern, how about waiting for a response from Port 9632 via netcat. Once that Port is responding, the supervisor can receive commands.
I have a while ! hab svc status 2>&/dev/null ; do sleep 5 ; done
thing going on, which closes the loop.
The things I don't like about that:
- The resource consumption that comes with polling. When I get to the point of having thousands of concurrent spin-ups, that'll have a measurable impact.
- for
sleep $n
, I lose up to $n seconds where the service was available, I just wasn't testing again for up to $n seconds. When a full end-to-end process takes less than 3 minutes, that's a significant fraction of the cycle time. If you're going to cycle several dozen times a day, that's significant.
As @mwrock suggests, here is what I do for plan build testing to make sure supervisor is available before loading services:
https://github.com/habitat-sh/core-plans/blob/master/consul/test.sh#L26-L35
I'd caution against a polling mechanism, since other applications that might open the socket before you, either maliciously or inadvertently could give false positives for applications thinking that the service is active.
The better option in this case is having the service send some form of state change to the OS to say it is ready to service requests. On Linux with SystemD and Windows, both have a way to take notifications and expose them to the non-executive layers of the system without these potential race conditions.
Alternatively, you can send notices on DBus on Linux, or other similar technologies on the BSDs or Windows. This would require tooling to subscribe to those IPC systems to collect this info, but that again, isn't overly difficult.
Finally, using OpenBSD-NetCat isn't always an option, since in higher security environments, nc and friends aren't allowed to be installed on the system.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.
This still affects us.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.