read_bloom_filter_index() take 2min for ontime

[youngsofun]

can we accelerate it？

cc @dantengsky

[dantengsky]

yes

• enable table cache bloom filter index will be cached (by col) if table cache is enabled

  but apparently, it only works after warming up

• allow more concurrency in pruning

https://github.com/datafuselabs/databend/blob/869344f106c69552ff61835c69da471f4de40f95/src/query/storages/fuse/src/pruning/pruning_executor.rs#L103-L106

btw, how many parts and rows for the test set?

[youngsofun]

btw, how many parts and rows for the test set?

202687555 rows.

how to show num of parts?

[BohuTANG]

SELECT * FROM fuse_snapshot('<database_name>', '<table_name>'); or explain with latest main branch.

[youngsofun]

[dantengsky]

got it , thanks!

[BohuTANG]

It would be better to create a bloom-filter index for STRING types only? cc @dantengsky

[BohuTANG]

@drmingdrmer is working on this to make it faster 🤗

[drmingdrmer]

Oh, man. Thanks for reminding me.

I'm trying to dig deeper to grasp what's going on inside the fuse table with the bloom filter.

I will give a refactoring proposal ASAP.

[BohuTANG]

@youngsofun Can you try with the main branch? I think we have some improvements with this patch: https://github.com/datafuselabs/databend/pull/7870

[youngsofun]

the problem is originally for Q10 on ontime. on a test cluster. and I found simplySELECT count(*) FROM default.ontime WHEREdepdel15=1; is abnormal: slow with many log about bloom filter

---

can not open test.datafusecloud.com now

[BohuTANG]

I think the time cost is small now, let's close, if you find the time is still long please re-open next time.