python-rocksdb add cache_index_and_filter

add cache_index_and_filter_blocks options, which is import when bloom filter

Jun 16 '17 14:06 lhztop

@lhztop

thanks for the PR.

it's great.

Can you add some test cases as well?

Jun 16 '17 15:06 twmht

Ok. now it's in test_options.py

Jun 19 '17 05:06 lhztop

Any reason this hasn't been pulled in?

Looks like it also makes it possible to expose bottommost_compression which would be nice to be able to configure (FB recommends LZ4 top, ZSTD bottom).

Feb 15 '18 10:02 gcp

@gcp

yup. some test cases failed, I am going to verify.

Feb 15 '18 11:02 twmht

Any news on this? According to https://github.com/facebook/rocksdb/issues/3216 this might be a solution to a memory leak issue I have.

Jun 26 '19 12:06 oersted

@oersted

yup. I am currently migrating all of the existing features to pybind11 branch.

Jun 27 '19 01:06 twmht

@oersted

so if we set cache_index_and_filter_blocks to true, then it can solve the memory leak problem?

Jun 27 '19 03:06 twmht

@twmht Actually it looks like it was an unrelated garbage collection issue, I was creating too many new small objects in my merge operation that weren't getting collected in time. Thanks anyways.

Jun 27 '19 08:06 oersted

Take some attention to https://github.com/facebook/rocksdb/blob/6.3.fb/include/rocksdb/table.h#L61-L71

I have several databases with different sizes: 30gb 250gb 400gb 1.2tb. I doesn't have any memory issue.

Jun 27 '19 09:06 iFA88

@oersted

I will add merge operations to pybind11 branch in a few days. after that You can try the branch to test if it has memory leak.

Jun 27 '19 11:06 twmht