rocksdb icon indicating copy to clipboard operation
rocksdb copied to clipboard

Calling DB::open() with create_missing_column_families=true is quadratic

Open sadderchris opened this issue 1 year ago • 0 comments

Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://groups.google.com/forum/#!forum/rocksdb or https://www.facebook.com/groups/rocksdb.dev

Expected behavior

Per the notes here, when calling DB::open() with create_missing_column_families=true, column family creation is batched and takes a reasonable amount of time when opening a fresh database with lots of new column families.

Actual behavior

Calling DB::open() with create_missing_column_families=true doesn't actually batch create new column families and instead is quadratic in the number of column families it needs to create.

https://github.com/facebook/rocksdb/blob/8b566964b8b30965ed5e70115921705379b750fe/db/db_impl/db_impl_open.cc#L2056-L2060

Steps to reproduce the behavior

  1. Set options.create_missing_column_families = true
  2. Call DB::Open() on an empty directory and request to open a few hundred column families
  3. Observe that the time needed to open is between 30s - several minutes

sadderchris avatar Jul 25 '24 18:07 sadderchris