PRIMARY_REGION not considered on cluster initialization

[davissp14]

Stolon has a keeper flag named can-be-master that was introduced by the following PR: https://github.com/sorintlab/stolon/pull/697

We use this flag in combination with the PRIMARY_REGION env var to restrict primary eligibility to a specific region.

Problem The problem is that Stolon's Sentinel does not consider this flag when assigning the primary on cluster initialization. During the cluster initialization process, the primary keeper is chosen at random.

More information can be found here: https://github.com/sorintlab/stolon/issues/840

Workaround The recommended approach as of right now, is to provision your cluster within the primary region BEFORE spinning up cross-region read-replicas. If you spin everything up at once, you will be at risk of an out-of-region member being assigned primary.

[mrkurt]

This is (currently) a non issue for flyctl pg create right? It just creates a cluster in a single region.

[davissp14]

@mrkurt Right, as far as provisioning goes this would only be an issue for manual deployments.

However, in the event the backend store is lost, the cluster will be forced to re-initialize. So it would seem there are additional opportunities outside of the initial provision where this could potentially bite someone.

Update

One notable scenario of concern would be someone transitioning their backend store from consul to etcd.