vault
vault copied to clipboard
hashicorp
parallel migration
This PR speed up keys migration using the vault operator migrate command.
Each key is migrated in a goroutine (the actual part that moves the data) and thus the blocking operation of copying data is handled separately from the part that iterates over objects in Vault. In addition, the number of parallel goroutines migrating the data can be controlled by setting the max_parallel option in the migration config.
Thanks to this change I manage to cut down Vault storage migration time between Consul and GCS from ~3h 30m to ~5m for around 90k of entries and max_parallel=100.
If that looks promising I will update the docs too.
@hsimon-hashicorp what is the prospect of getting this one merged ?
I'll see what I can do! I've asked for a review from our engineer on rotation - please feel free to ping me again if need be, as well. :)
@gites are you still active, and would you be willing on answering the questions?
@gites is attempting to deploy a commit to the HashiCorp Team on Vercel.
A member of the Team first needs to authorize it.
![vercel[bot] avatar](https://avatars.githubusercontent.com/in/8329?v=4 "Published by a pub.dev verified publisher"){kind=small}
Hello! I would like to finish this PR(it looks like almost everything is already here, there are little things left). @gites what do you think? and thanks for all this work!
@gites and @msheldyakov There are some comments and questions that we would like to sort out before merging the PR. Would you be interested to respond/address those comments?
@gites I tried to help you out with some things if you've been too busy to get back to this, appreciate it's been a long lived affair.
https://github.com/gites/vault/pull/1
Hi folks, are we still working on this? Otherwise, our engineers may be able to fork it and make some progress and get it merged. Thanks in advance!
Hi folks, are we still working on this? Otherwise, our engineers may be able to fork it and make some progress and get it merged. Thanks in advance!
Hey @hsimon-hashicorp, it may well be that the original contributor just hasn't the time now to continue this and it would be a shame to see their efforts wasted. I'm happy to push my fork of @gites' work into a HashiCorp branch so we can try and wrap it up.
I'll leave this comment up for a few days in case @gites and other co-authors want to finish it themselves, and failing a response, or a response to suggest we 'go ahead' I think we should do just that. :)
Hi folks, are we still working on this? Otherwise, our engineers may be able to fork it and make some progress and get it merged. Thanks in advance!
Hey @hsimon-hashicorp, it may well be that the original contributor just hasn't the time now to continue this and it would be a shame to see their efforts wasted. I'm happy to push my fork of @gites' work into a HashiCorp branch so we can try and wrap it up.
I'll leave this comment up for a few days in case @gites and other co-authors want to finish it themselves, and failing a response, or a response to suggest we 'go ahead' I think we should do just that. :)
Oh I see that someone else already asked and was granted permission to finish this off (in the comments above) https://github.com/hashicorp/vault/pull/8857#issuecomment-1313912890. Since nothing is happening on the PR I will start trying to move into 'into' HashiCorp.