Cannot serially `hab config apply` key/value pairs into running service config

[bixu]

I'm working on a package (bixu/certbot) that provides Certbot-generated LetsEncrypt certificates automatically. In order to allow other services in the ring to retrieve updated certificates, load these certs and restart as needed, it's necessary to inject the certs as key values into the gossiped config shared by members of the ring.

Unfortunately, I'm seeing behavior where only one key at a time is stored in the gossiped config if I create and inject keys serially. Here's a simple example to demonstrate this behavior:

• [ ] The OS: macOS 10.14.3 (18D109), Docker studio
• [ ] Debug/backtrace of the command:

$ docker --version
Docker version 18.09.2, build 6247962

hab studio enter

[1][default:/src:0]# hab svc load core/nginx

[2][default:/src:0]# # type test_injection
test_injection is a function
test_injection () 
{
    export RUST_BACKTRACE=1 RUST_LOG=debug
    hab pkg install core/curl --binlink;
    hab pkg install core/jq-static --binlink;
    while ! hab svc status core/nginx; do
        sleep 1;
    done;
    hab svc load core/nginx;
    curl --silent localhost:9631/services/nginx/default | jq '.cfg';
    echo 'key1 = "foo"' | hab config apply nginx.default "$(date +%s)";
    sleep 3;
    curl --silent localhost:9631/services/nginx/default | jq '.cfg';
    echo 'key2 = "bar"' | hab config apply nginx.default "$(date +%s)";
    sleep 3;
    curl --silent localhost:9631/services/nginx/default | jq '.cfg'
}

[3][default:/src:0]# test_injection
[2019-03-06T10:33:21Z DEBUG habitat_common::ui] UI { shell: Shell { input: InputStream { isatty: true }, out: OutputStream { coloring: Never, isatty: true, is_colored(): false, supports_color(): true }, err: OutputStream { coloring: Never, isatty: true, is_colored(): false, supports_color(): true } } }
[2019-03-06T10:33:21Z DEBUG hab] clap cli args: ["hab", "pkg", "install", "core/curl", "--binlink"]
[2019-03-06T10:33:21Z DEBUG hab] remaining cli args: []
[2019-03-06T10:33:21Z DEBUG hab::config] No CLI config found, loading defaults
[2019-03-06T10:33:21Z DEBUG habitat_common::command::package::install] install key_cache_path: /hab/cache/keys
[2019-03-06T10:33:21Z DEBUG habitat_http_client::ssl] Setting CA file for SSL context to: /hab/pkgs/core/cacerts/2018.12.05/20190115014206/ssl/cert.pem
[2019-03-06T10:33:21Z DEBUG habitat_http_client::api_client] Client socket timeout: 120 secs
[2019-03-06T10:33:21Z DEBUG habitat_http_client::api_client] User-Agent: hab/0.76.0/20190227200538 (x86_64-linux; 4.9.125-linuxkit)
» Installing core/curl
☁ Determining latest version of core/curl in the 'stable' channel
...DOWNLOAD DEBUG OUTPUT TRIMMED...
★ Binlinked curl-config from core/curl/7.63.0/20190201004909 to /bin/curl-config
» Installing core/jq-static
☁ Determining latest version of core/jq-static in the 'stable' channel
...DOWNLOAD DEBUG OUTPUT TRIMMED...
★ Binlinked jq from core/jq-static/1.10/20160909011845 to /bin/jq
{
  "events": {
    "worker_connections": 1024
  },
  "http": {
    "keepalive_timeout": 60,
    "listen": {
      "port": 80
    },
    "sendfile": "on",
    "tcp_nodelay": "on",
    "tcp_nopush": "on"
  },
  "worker_processes": 4
}
[2019-03-06T10:33:24Z DEBUG habitat_common::ui] UI { shell: Shell { input: InputStream { isatty: false }, out: OutputStream { coloring: Never, isatty: true, is_colored(): false, supports_color(): true }, err: OutputStream { coloring: Never, isatty: true, is_colored(): false, supports_color(): true } } }
[2019-03-06T10:33:24Z DEBUG hab] clap cli args: ["hab", "config", "apply", "nginx.default", "1551868404"]
[2019-03-06T10:33:24Z DEBUG hab] remaining cli args: []
[2019-03-06T10:33:24Z DEBUG hab::config] No CLI config found, loading defaults
» Setting new configuration version 1551868404 for nginx.default
Ω Creating service configuration
[2019-03-06T10:33:24Z DEBUG tokio_reactor::background] starting background reactor
[2019-03-06T10:33:24Z DEBUG tokio_reactor] dropping I/O source: 0
↑ Applying via peer 127.0.0.1:9632
[2019-03-06T10:33:24Z DEBUG tokio_reactor] dropping I/O source: 0
★ Applied configuration
{
  "events": {
    "worker_connections": 1024
  },
  "http": {
    "keepalive_timeout": 60,
    "listen": {
      "port": 80
    },
    "sendfile": "on",
    "tcp_nodelay": "on",
    "tcp_nopush": "on"
  },
  "key1": "foo",
  "worker_processes": 4
}
[2019-03-06T10:33:27Z DEBUG habitat_common::ui] UI { shell: Shell { input: InputStream { isatty: false }, out: OutputStream { coloring: Never, isatty: true, is_colored(): false, supports_color(): true }, err: OutputStream { coloring: Never, isatty: true, is_colored(): false, supports_color(): true } } }
[2019-03-06T10:33:27Z DEBUG hab] clap cli args: ["hab", "config", "apply", "nginx.default", "1551868407"]
[2019-03-06T10:33:27Z DEBUG hab] remaining cli args: []
[2019-03-06T10:33:27Z DEBUG hab::config] No CLI config found, loading defaults
» Setting new configuration version 1551868407 for nginx.default
Ω Creating service configuration
[2019-03-06T10:33:27Z DEBUG tokio_reactor::background] starting background reactor
[2019-03-06T10:33:27Z DEBUG tokio_reactor] dropping I/O source: 0
↑ Applying via peer 127.0.0.1:9632
[2019-03-06T10:33:27Z DEBUG tokio_reactor] dropping I/O source: 0
★ Applied configuration
{
  "events": {
    "worker_connections": 1024
  },
  "http": {
    "keepalive_timeout": 60,
    "listen": {
      "port": 80
    },
    "sendfile": "on",
    "tcp_nodelay": "on",
    "tcp_nopush": "on"
  },
  "key2": "bar",
  "worker_processes": 4
}
```	
- [ ] Current Habitat environment variables:
  - [ ] a. before starting the studio

$ env | grep HAB | grep -v "TOKEN|PASSWORD|SECRET" HAB_NOCOLORING=true HAB_HART_COMPRESSION_LEVEL=0 HAB_NONINTERACTIVE=true

  - [ ] b. inside the studio

[1][default:/src:0]# env | grep HAB | grep -v "TOKEN|PASSWORD|SECRET" HAB_ORIGIN=bixu HAB_NOCOLORING=true

[bixu]

cc @baumanj ( see also https://forums.habitat.sh/t/unable-to-inject-multiple-key-values-sequentially-via-hab-config-apply/1040 )

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.