keeper: support --wal-dir

For those who run Postgres with a WAL directory symlinked to a separate disk (often for performance reasons) this commit adds a --wal-dir flag to the keeper.

The symlink to a new WAL disk needs to be created as part of the database initialisation step, and also whenever we clobber the database, such as when taking a pg_basebackup (if pg_rewind is disabled/unavailable). By passing this directory as an argument to initdb and pg_basebackup, we can cover all the scenarios that the keeper erases and recreates the data directory, ensuring the symlink is always present.

Opening this PR as an RFC: I'm not entirely familiar with the stolon codebase, so it might be that this is insufficient or better implemented elsewhere. Going to test this with our installation of stolon today to see if it works in practice.

We've been running this for a week now in several environments and everything looks fine. No rush as we've deployed this via our fork, but it's now ready for review @sgotti.

@lawrencejones Thanks for your PR! Overall it LGTM. I'd like to have some integration tests that exercise the use of --wal-dir to check if everything works. For example to test that the resync works also if the keepers have different wal-dir paths (or if one keeper has wal-dir defined while another doesn't have it) and that this works also with pg_rewind.

That makes a lot of sense- will get to this next week.

Hey @sgotti - some tests have been added to this branch. Can you take another look? Thanks!

Hey @sgotti- can we help on this at all?

@sgotti Would you mind taking another look at this? I did this same thing on my own fork back in March. It would be excellent to have first-class support for this. Thanks!

@sgotti Bump. Can we get this upstream please?

@lawrencejones I have the rebased, squashed commit in my fork working well. If you don't have time to rebase, I can submit a new PR with your commit that is rebased and ready to go.

https://github.com/IronNetCybersecurity/stolon/commit/01d0aaa6995c0f6834d247c702133360b7d0be35

Hey @jnicholls, we've been running with this for a long time at GoCardless without issue. I'm actually a maintainer on this project now, so I think if your rebase fixes these conflicts then we should get this merged.

@lawrencejones That's fantastic, thanks. If I can help, how would you like me to proceed? I can submit a fresh PR from my fork, either with the original 3 commits unsquashed or not (whichever you prefer, you can squash them in the merge commit in GitHub). Or would you prefer to take care of the conflict resolution on your fork and merge this particular PR?

@lawrencejones n/m, I see the new commit :)

I stole the commit from your fork, after starting to deal with the conflicts and realising what a pain that was. Thanks for dealing with it for me!

I'm going to ask that one of my team give this PR a once-over just to make sure we haven't missed anything, but after that's good then we can get it merged.

@lawrencejones Note that my commit contains go.sum updates as well for the checksum of the root of a bunch of dependencies (go v1.13.6).

Thanks for the tip, going to push an updated go.sum now.

Hi, I would like to have this feature too. But this PR seems to be stale since over a year. Can I adopt and finalise this PR, or can I issue a new PR? Let me know how I can help to bring this feature into stolon...

@sebasmannem Feel free to open a new PR (PRs can't be moved to another user/org since they're tied to an user/org fork branch)