Concurrent stacker builds

[ejholmes]

trafficstars

We have a fairly small team of people that run stacker right now, so it's easy to ask someone if they're doing something in an environment, but I think it's worth talking about what it would look like if we had 5, 10, 20 people running stacker build in a continuous delivery environment.

Let's consider the following scenario:

We have the following stacks, which have dependencies on other stacks:

1. "Bob" makes a change to the appA stack, and commits it to master as a27e0e2.
2. "Sue" makes a change to the appB stack, and commits it to master as e08f9ec a couple seconds after Bob.
3. "John" makes a change to the appC stack, and commits it to master as 5dbfc38 couple seconds after Sue.

Now, my question is, what's the fastest way to perform these updates?

A naive solution would be to simply obtain a lock on the entire stacker build, however, that's unecessary since some changes are mutually exclusive. In theory, both Bob's and Sue's changes should be able to execute in parallel, but John's change needs to wait on Sue's change:

It seems like we could:

1. Obtain a lock when we update a stack.
2. If trying to update the stack results in a noop, release the lock.
3. If the stack updates, hold the lock until the stacker build completes.

With this both Bob and Sue's changes would execute in parallel. Sue's change would obtain a lock on the appB stack. John's change would try to obtain a lock on the appB stack, but block until Sue's change is complete, then continue with a noop change to appB and continue updating appC.

Thoughts? Where does this break down?

[phobologic]

#65 is likely a duplicate of this

[ttaub]

How are you actually obtaining a lock on a stack? What would the implementation of a lock look like?

[ejholmes]

@ttaub probably dynamo so it can be shared easily, but could be anything that can implement a lock(), unlock() interface. Empire has a similar concept to synchronize many concurrent deployments. It uses postgres advisory locking: https://github.com/remind101/empire/blob/master/scheduler/cloudformation/cloudformation.go#L570-L592

I'm mostly curious about what algorithms there are for synchronizing multiple separate concurrent executors walking a graph. I'm sure this is a solved problem, and there's a paper on this subject somewhere...