dolt icon indicating copy to clipboard operation
dolt copied to clipboard

`dolt` cli commands take a long time for large databases

Open VinaiRachakonda opened this issue 2 years ago • 1 comments

With really large databases (couple hundred gb) that contain large indexes, the dolt cli takes substantially longer. This is due to the fact dolt loads indexes into memory in order to properly compute answers for functions like dolt status.

This behavior has been verified with the Bitcoin database and FBI-NBIRS.

I'm still investigating potential optimizations, but for now it is recommend for users to use dolt sql shell for consistent access to version control features via the Dolt SQL functions.

This also relates to running dolt log on which takes a while to manifest a large commit chain

VinaiRachakonda avatar Jun 16 '22 17:06 VinaiRachakonda

Known. The commit graph needs to be loaded into memory for the command line to execute. This can take a long time.

timsehn avatar Aug 03 '22 23:08 timsehn