Write migration guide

[JanMa]

Now that we have a first release, we also need to think about what users of upstream Vault need to do in order to migrate to OpenBao. The migration guide should cover a migration from vault 1.14.x, which should mostly be a drop in replacement, as well as a migration from later releases to which we are not necessarily API or seal compatible.

[munir131]

@JanMa Is it done? I found backup or snapshot restore. Does that going to work?

[JanMa]

Hey @munir131 so far nobody wrote an explicit migration guide. If you're still on Vault 1.14 and are using one of our supported storage backends, you can most likely perform an in-place upgrade to OpenBao. If you're running a later version using bao operator raft snaphot restore should also work

[jficz]

I'll rewrite my notes of a successful migration to a something more guide-like, unless someone has already started in which case we can exchange the notes :)

[Jasper-Ben]

If you're running a later version using bao operator raft snaphot restore should also work

We are currently doing a test migration from vault v1.20.1 via raft snapshot restore and so far it looks good! We were able to restore and unseal in openbao and all secrets seem to be available (we still have to do some sanity checks)

[Jasper-Ben]

Ok, we hit a first minor issue during testing. In our vault we had an AWS authentication method configured. This authentication method was broken after restoring the snapshot and could not be removed:

Error disabling auth method at aws/: Error making API request.

URL: DELETE https://openbao.xxx/v1/sys/auth/aws
Code: 400. Errors:

* cannot write to storage during setup

After some research I discovered that the auth-aws plugin is not a builtin in openbao. This is the case for many plugins that are shipped with vault (54 plugins in vault vs 27 in openbao).

~~For me this raises the question whether it makes sense for openbao to include the same builtin plugins as vault, which IMO would make it feel more like a drop-in replacement. Is this something that is already planned for the future @JanMa?~~

Found https://github.com/orgs/openbao/discussions/64 since, which explains the direction with plugin support

[Jasper-Ben]

For migration guides, it might be useful to point out that previously used plugins in vault might not work in openbao and that adjustments might become necessary before attempting the migration

[cipherboy]

Tbh, I'm not sure snapshots bring you much. But if you can interact with the plugin catalog, you should be able to register the external plugin under the existing name in storage and restart. That was our intended migration scenario.

At any rate, future work around declarative plugin initialization was proposed but nobody was interested in implementing it. If you're interested, happy to work with you on the design and implementation!

[Jasper-Ben]

Tbh, I'm not sure snapshots bring you much.

Can you elaborate please? Do you mean in general or in the context of plugins. Generally, from what we can see so far from our testing, it seems to be possible to migrate from vault to openbao using a raft snapshot, which to my understanding would be the only option for migrating a newer version of vault.

In any case, regardless of whether migration is done via snapshot or in-place upgrade, the availability (or lack thereof) of builtin plugins in openbao compared to vault can in certain scenarios (such as ours) lead to issues when attempting to use openbao as a drop-in replacement. This isn't necessarily a big deal, but if we formulate a migration guide, it should mention that only a subset of vault plugins are supported in openbao and that additional steps (e.g. installing plugins in openbao or migrating to a supported storage) might be necessary before attempting a migration.

[cipherboy]

@Jasper-Ben I think @JanMa has actually started on declarative plugin registration.

Can you elaborate please? Do you mean in general

Yes, in general, especially:

would be the only option for migrating a newer version of vault.

I think the one case where this would help is if we have substantially mismatched Raft/bbolt versions. But in general, no application/secret data is migrated differently as a result of snapshot restore versus drop-in binary swaps. It might be more resilient to downgrading bbolt versions I believe (if Vault were to have a newer bbolt dependency version than OpenBao), but wouldn't necessarily improve data migration like in this case.

Your alternative might be to first add the external plugin to Vault (overriding the builtin plugin, with our plugin name) and then swapping the binary to OpenBao. This would let OpenBao come up with the external plugin already registered for that plugin and so should succeed. I believe builtin plugin shadowing is an explicitly tested use case as well.

My 2c.

[Jasper-Ben]

Your alternative might be to first add the external plugin to Vault (overriding the builtin plugin, with our plugin name) and then swapping the binary to OpenBao. This would let OpenBao come up with the external plugin already registered for that plugin and so should succeed. I believe builtin plugin shadowing is an explicitly tested use case as well.

Ah, good to know that this is supported!

The path we opted for (since we didn't want to touch production during evaluation) was to replicate our vault deployment (raft snapshot create -> raft snapshot restore), disable the aws authentication in the replica (and migrate the seal from auto-unseal to shamir, since we are also doing a cross-AWS account migration at the same time, but that is another story), and create a new snapshot from the vault replica and import that into openbao, thus avoiding the broken auth method.

Next step for us is to re-enable the aws auth method in openbao (which currently seems to be semi-blocked by https://github.com/openbao/openbao-plugins/issues/28)

[jficz]

these are good points that should be added to the guide, I'll add them to the PR but suggestions/patches are welcome ;)