`hab svc status` never returns

[bixu]

We've recently run into an issue where

hab svc status

never returns, which wreaks havoc on our Terraform bootstrapping code (we use the return code of the command to check that the Supervisor is running).

• [x] The OS (including version) where you are running any of the Habitat commands:

$ uname -a
Linux staging-fullstack-0 4.4.0-1065-aws #75-Ubuntu SMP Fri Aug 10 11:14:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/issue
Ubuntu 16.04.5 LTS \n \l
# hab --version
hab 0.62.1/20180905002037

• [x] Debug/backtrace of the command you are trying to run:

$ sudo RUST_LOG=debug RUST_BACKTRACE=1 hab svc status
DEBUG 2018-09-11T12:47:01Z: habitat_common::ui: UI { shell: Shell { input: InputStream { isatty: true }, out: OutputStream { coloring: Auto, isatty: true, is_colored(): true,
supports_color(): true }, err: OutputStream { coloring: Auto, isatty: true, is_colored(): true, supports_color(): true } } }
DEBUG 2018-09-11T12:47:01Z: hab: clap cli args: ["hab", "svc", "status"]
DEBUG 2018-09-11T12:47:01Z: hab: remaining cli args: []
DEBUG 2018-09-11T12:47:01Z: hab::config: No CLI config found, loading defaults
DEBUG 2018-09-11T12:47:01Z: tokio_reactor::background: starting background reactor
DEBUG 2018-09-11T12:47:01Z: tokio_reactor: loop process - 1 events, 0.000s
DEBUG 2018-09-11T12:47:01Z: tokio_reactor: loop process - 1 events, 0.000s
^C

• [x] Current Habitat environment variables where the hab command or supervisor is running: No HAB variables are set except for HAB_AUTH_TOKEN.

cc: @smartb-pair

[christophermaier]

@bixu / @smartb-pair Is the Supervisor currently otherwise occupied (e.g., running a long-running init hook for some service)? If so, this is (unfortunately) expected behavior right now, until we get the hooks executing outside the main Supervisor thread. That work is going on right now, though.

(I know this sucks... until it's fixed, you might be able to get a more useful signal by probing the HTTP gateway, if that's possible for your setup.)

If the Supervisor isn't doing something else, though, then that's interesting and unexpected, and I'd love to know more about your specific setup.

[bixu]

@christophermaier, that makes much more sense then. I think what's happening is that we have init hooks that are crashing because of an oversight in how we communicate between two services. I'm working on fixing that right now.

I still think this should be a bug because the behavior is so mysterious (took me a few hours to guess that it might have to do with services restarting).

[christophermaier]

@bixu Yup, agreed... we'll keep this open in case others end up searching for it. In the mean time, the main issue to track is #5329.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.