lakeFS icon indicating copy to clipboard operation
lakeFS copied to clipboard

Published 20 hours ago • verified publisher icon treeverse

Benchmark kv migration

Open itaiad200 opened this issue 2 years ago • 1 comments

At least 1M objects in staging

itaiad200 avatar Jul 31 '22 08:07 itaiad200

A possible optimization hides in the comments of #3648

itaiad200 avatar Jul 31 '22 10:07 itaiad200

It took 9m to migrate 1M objects from DB to KV locally. Assuming the number of entries which are not under staging is much smaller than that (<<100k) and that the staging area can be committed to speed up migrations, I believe these numbers are good enough.

itaiad200 avatar Aug 14 '22 11:08 itaiad200