Julien

Close! there is a second race.

``` goroutine 2289585 [running]: runtime.throw(0x16e8c4d, 0x21) /home/roidelapluie/godist/go/src/runtime/panic.go:617 +0x72 fp=0xc00031cbc0 sp=0xc00031cb90 pc=0x42f982 runtime.mapaccess1(0x14683c0, 0xc000408ff0, 0xc00031ce18, 0x19f9d20) /home/roidelapluie/godist/go/src/runtime/map.go:413 +0x265 fp=0xc00031cc08 sp=0xc00031cbc0 pc=0x40fbe5 github.com/purpleidea/mgmt/engine/graph.(*Engine).Worker(0xc00022c460, 0x7f74a99d4f08, 0xc00045eb40, 0x0, 0x0) /home/roidelapluie/go/src/github.com/purpleidea/mgmt/engine/graph/actions.go:260 +0x116 fp=0xc00031cf60 sp=0xc00031cc08 pc=0xc949b6...

Due to the clustering nature of alertmanager, it would not be possible to use retry after, which is why I am not sure we should accept this pull request

Yes. However, they are cloud services. For alertmanager webhooks, we expect them to be written specifically for alertmanager, therefore never return 429's.

Prometheus only supports environment variables in very specific areas of the configuration.

The way to deal with this is to put env variables into files and read from files. A lot of useful fields already support that. What is your use case?

It's available but .Alerts is a slice, not a single alert.

I am puzzled on this one. It seems that you are using gossip instead of persistent disk.

Empty labels are valid in Prometheus and alertmanager. cc @beorn7 WDYT here?

yeah, "" will be dropped by shell for most users anyway. I suggest we consider it as an empty value.