Plugin support

[bmorton]

Deployster itself is pretty simple and scoped, but being able to take action after Deployster has done certain things would be pretty awesome. Some examples of things that Deployster likely won't support, but would be awesome to implement as a plugin:

• Notify Hipchat of a new deploy
• Log deploys to a database
• Trigger an integration test build after deployment has finished

Given that these things likely don't belong in the main repository, it'd be extremely useful to implement a way for Deployster to emit events after it has completed certain tasks so that listener plugins can take action on that data. Suggestions on implementations of this (or examples where this has been done well other places) are appreciated.

Examples of some events we could emit:

• Deploy started
• Deploy completed
• Deploy failed
• Task started
• Task completed

[bmorton]

Continuing to look into this and need to experiment with dbus to see if this is the right way to emit events. This discussion has a way to mount access to the host's systemd in a Docker container, hopefully providing access to the host's dbus for the deployster container.

This is the command from that discussion:

docker run -it -v /var/run/dbus:/var/run/dbus -v /run/systemd:/run/systemd -v /usr/bin/systemctl:/usr/bin/systemctl -v /etc/systemd/system:/etc/systemd/system debian:jessie /bin/bash 

Here's a golang library for working with dbus: https://github.com/godbus/dbus

[bmorton]

Starting to prototype this up now using etcd. I started by researching dbus more, but I'm not convinced it'll do what we need it to do and it is pretty difficult to work with. It also limits us to running plugins on the same machine as deployster.

Here's the approach that I'm going with at the moment:

• Deployster publishes JSON payloads to etcd at /deployster/events/{timestamp} with a TTL of 1 day by default, but configurable.
• deployster-plugin package implements simple way to add handlers to watch for these events and keep track of the last event seen so that it can replay events while it was offline, up to the TTL, if necessary.
• Third party developers create daemons (like deployster-yammer) that use the deployster-plugin package to watch for events and act on them.

[ivanvanderbyl]

I'm not sure shoehorning etcd in to being an event bus is a good idea. We should maybe look at Consul for this, or a proper message bus.

[ivanvanderbyl]

I've been reading about mesosphere/marathon recently, they have a pretty cool feature set, and a rather similar goal to us, albeit for much large scale, and for Apache Mesos instead of CoreOS.

Here's their docs on event bus https://mesosphere.github.io/marathon/docs/event-bus.html

[bmorton]

Yeah, I hear you. Thanks for the sanity check. I've been spinning my wheels trying a few routes on this and the etcd one is the first one I made real progress on, so it felt good. I have felt all along that using etcd in this way feels dirty though.

I'm really trying to avoid having to bring in anything extra for this, so these are some other options that are still possible:

• Emit events to the journal and plugin implementors can watch the systemd journal. I think plugins would need to run on the same machine as deployster in this case.
• Figure out how to add a notification stream to dbus, but I think our use case is backwards from what dbus does. I think that dbus allows you to listen for other things to notify you and wouldn't handle fanning out to lots of subscribers.
• Allow HTTP callbacks to be registered and have plugins just be web services. I think this is the marathon approach you linked above. I'm leaning towards this option.

[ivanvanderbyl]

Another option, slightly more complicated, but more flexible for the subscribers, look at NSQ http://nsq.io/overview/design.html