Allow vulcand to be substituted with other alternatives

[bmorton]

Currently, we assume that vulcand is installed and use that to accomplish zero downtime deploys. We should allow different solutions here, including things like Nginx or HAProxy. Both alternatives would probably need a special container created for them that uses something like confd to watch for changes in etcd and reload accordingly.

This should also provide a path for us to support the latest version of vulcand, as well.

Steps

• [ ] Extract out vulcand configuration to a proper sidekick unit
• [ ] Add runtime option for choosing the reverse proxy layer
• [ ] Add vulcand 0.8.0 sidekick
• [ ] Add nginx sidekick
• [ ] Add haproxy sidekick

[ivanvanderbyl]

So I've hacked together a few ideas this morning around this, and came up with some questions:

In my first approach, I implemented a service which runs as a sidekick called deployster-sidekick, it accepts a bunch of flags to configure itself, but generally it knows the process it's bound to. One of the options it takes is --upstream, which can be vulcan-07, or vulcan-08, which both have differing APIs. Other options could be implemented to support nginx or haproxy. The sidekick process handles talking to etcd to set and remove the necessary vars depending on the upstream chosen.

My original thinking for implementing it as a separate service was to allow for more complicated logic around vulcan-08, which could allow for idempotently setting up the front-end as well, if it isn't defined. This could allow us to pass an initial config to deployster for configuring the proxy. Maybe as a separate endpoint, or as part of /deploy.

The second approach is to build all of the above logic in to deployster and have it write sidekick units which call out to etcdctl explicitly. This is the simplest solution, but it is limited.

Which do you think makes the most sense?

[bmorton]

I'll bounce a couple questions back at you:

How does deployster/deployster-sidekick get enough information to setup the frontend? It'd have to know the routing logic you want to use for that. For now, I had kept that outside the responsibility of deployster, but I'd love to hear ideas about this.

How does deployster interact with deployster-sidekick? I'm having a little trouble visualizing it, so if you could walk me through what'd happen on a deploy, that'd be super helpful for me.

[ivanvanderbyl]

I have a couple of ideas around this, I'm open to suggestions on what fits the direction of deployster more.

Firstly, deployster-sidekick is launched as a sidekick job next to the unit you're deploying. My initial design was to have its only interaction be from deployster when writing the unit file, and pass everything it needs to know as ENV vars or args. This could also be done with an IPC mechanism. It will probably be launched on another node so it could also just talk to the deployster API to get the information it needs on launch.

As for where deployster gets the routing logic from, it could be supplied in the deploy payload when doing a create, or from another endpoint designed for setting up routing.

The more I think about this though, I think it would make sense for the sidekick to exclusively handle the registration and de-registration only. This can be done by simply calling out to etcdctl as per usual.

It would make more sense for a separate worker process/goroutine to handle setting up routing. This would work well with an endpoint for setting up routing. This also gives a lot of flexibility to the client. If this could be designed in a way in which neither a routing request or a deployment request are dependent on each other, it would greatly simplify getting started.

In an ideal world, I'd imagine getting started with Deployster being as simple as:

1. Launching a CoreOS instance with the deployster.service configured, and a vulcan.service, haproxy.service, or nginx.service. When the deployster service comes online it configures itself with the proxy and you now have a unified endpoint for talking to it.
2. You request a deploy of a docker container from a repository somewhere. The deploy declares whether this container exposes any ports, and what domain it should be routed to. If the container does expose ports it automatically gets routed in vulcan/ha/nginx etc. If it doesn't, i.e. it's a worker, then it can figure that out itself.
3. Upon re-deploy, if a service changes ports/domains then all previous instances are replaced with the updated configuration and routing logic updated.

[bmorton]

:+1: to having deployster only handle registration/de-registration. I've thought about building some kind of UI or something for vulcand independently of this project for making setting that stuff up easy. I also just recently found the vctl command which has simplified some stuff for me too (probably makes sense to keep using etcdctl for deployster though).

I like the steps at the bottom, but I'm going to try to restate the deployster-sidekick stuff to make sure I understand:

1. Deploy is triggered as HTTP request to deployster
2. Deployster stages the sidekick units for each service unit (or is this where it hands off the responsibility to something else?)
3. Deployster stages each service unit
4. Deployster toggles each service unit to running
5. Sidekick units start automatically because they are bound to the service units

For the staged sidekick units, it'd be nice if we had a helper utility (sidekicker?) that abstracted away the complexities of registration/de-registration for each different thing that we support. Something like this:

[Unit]
Description=Sidekick for carousel:ef32bca:2006.01.02-15.04.05@%i
BindsTo=carousel:ef32bca:2006.01.02-15.04.05@%i.service
After=carousel:ef32bca:2006.01.02-15.04.05@%i.service

[Service]
ExecStartPre={install sidekicker if necessary}
ExecStart=/usr/bin/sidekicker discover carousel --version=ef32bca --timestamp=2006.01.02-15.04.05 --instance=%i --host={?} --port={?} --load-balancer=vulcand --ttl=60s
ExecStop=/usr/bin/sidekicker remove carousel --version=ef32bca --timestamp=2006.01.02-15.04.05

[X-Fleet]
MachineOf=carousel:ef32bca:2006.01.02-15.04.05@%i.service

[bmorton]

That unit file is mostly copied from the "Run a simple sidekick" section of this CoreOS guide.

[ivanvanderbyl]

Ah yes sorry for the confusion, deployster-sidekick is the helper utility in your example. The thinking was that instead of installing it on the host as you put it, we could run it as another docker container (which also helps with versioning it).

[bmorton]

No worries! I like this. I think keeping it in a docker container is a good idea too. Are you interested in doing either side of this work? I have a rough implementation started of starting to extract out the template into something a little more manageable. I can push that up and we could go from there.

[ivanvanderbyl]

Yeah I'm happy to take on the helper stuff. I've already made a start last weekend: https://github.com/bmorton/deployster/compare/master...ivanvanderbyl:feature/sidekicks

As for templates, I think we should take the Deis approach and just have them as standalone files, then add them to the docker image on build.

[bmorton]

Awesome :+1:

[bmorton]

It looks like you've got a good chunk of this whole thing handled, so I'll let you handle this first and then we can figure out if we want to change around templates to match the Deis approach after this hits master.