habitat icon indicating copy to clipboard operation
habitat copied to clipboard

Feature Request: Log to stdout when supervisor starts successfully

Open smford22 opened this issue 6 years ago • 9 comments

Right now, hab-sup logs "Starting http-gateway on..." as the last line as it's starting up. Can we get a log message to the effect of "supervisor started successfully"?

It's in support of this pattern:

# that long, period, but just in case.
timeout 300s tail -f /var/log/messages | grep -m1 “Starting http-gateway”
# Give it one additional second, as we don’t have a post-started message.
sleep 1
/bin/hab svc load ${base_pkg} --channel stable --strategy at-once --force```

(Trying to deal with a race condition)

smford22 avatar Aug 09 '18 12:08 smford22

Other (potentially better, if less portable) solutions would be supporting SystemD's notify or dbus service types. At that point, I can simply ask SystemD to block me until the supervisor service comes up.

mikemol avatar Aug 09 '18 12:08 mikemol

Definitely agree that integration with systemd is good, but as an alternative to the above pattern, how about waiting for a response from Port 9632 via netcat. Once that Port is responding, the supervisor can receive commands.

mwrock avatar Aug 09 '18 13:08 mwrock

I have a while ! hab svc status 2>&/dev/null ; do sleep 5 ; done thing going on, which closes the loop.

The things I don't like about that:

  • The resource consumption that comes with polling. When I get to the point of having thousands of concurrent spin-ups, that'll have a measurable impact.
  • for sleep $n, I lose up to $n seconds where the service was available, I just wasn't testing again for up to $n seconds. When a full end-to-end process takes less than 3 minutes, that's a significant fraction of the cycle time. If you're going to cycle several dozen times a day, that's significant.

mikemol avatar Aug 09 '18 14:08 mikemol

As @mwrock suggests, here is what I do for plan build testing to make sure supervisor is available before loading services:

https://github.com/habitat-sh/core-plans/blob/master/consul/test.sh#L26-L35

predominant avatar Aug 15 '18 23:08 predominant

I'd caution against a polling mechanism, since other applications that might open the socket before you, either maliciously or inadvertently could give false positives for applications thinking that the service is active.

The better option in this case is having the service send some form of state change to the OS to say it is ready to service requests. On Linux with SystemD and Windows, both have a way to take notifications and expose them to the non-executive layers of the system without these potential race conditions.

Alternatively, you can send notices on DBus on Linux, or other similar technologies on the BSDs or Windows. This would require tooling to subscribe to those IPC systems to collect this info, but that again, isn't overly difficult.

Finally, using OpenBSD-NetCat isn't always an option, since in higher security environments, nc and friends aren't allowed to be installed on the system.

greeneg avatar Aug 17 '18 01:08 greeneg

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.

stale[bot] avatar Apr 05 '20 00:04 stale[bot]

This still affects us.

mikemol avatar Apr 05 '20 14:04 mikemol

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.

stale[bot] avatar Aug 13 '22 19:08 stale[bot]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.

stale[bot] avatar Oct 15 '23 15:10 stale[bot]