rust-rocksdb icon indicating copy to clipboard operation
rust-rocksdb copied to clipboard

Published 20 hours ago • verified publisher icon tikv

PCP-27: level compaction skip some sst to reduce write amplification

Open zhangjinpeng87 opened this issue 5 years ago • 4 comments

Description

Ln sst1[a...b e...f] Ln+1 sst2[a...b] sst3[c...d] sst4[e...f] Assume a level compaction picks sst1, sst2, sst3 and sst4 as source ssts, we can see sst3's key range is a gap in sst1, but their ranges overlaps. And sst3's content will be unnecessarily re-written in level n+1, this will cause write amplification. In TiDB the gap between index keys and row keys for the same row is broad, and will cause extra write amplification.

Difficulty

  • Hard

Score

  • 3600

Mentor(s)

  • @zhangjinpeng1987

Recommended Skills

  • Algorithm

zhangjinpeng87 avatar Nov 04 '19 08:11 zhangjinpeng87

@huachaohuang Seems you have finish this issue https://github.com/facebook/rocksdb/pull/6021, Do you have interest to participate our performance challenge match? https://pingcap.com/community-cn/tidb-performance-challenge/

zhangjinpeng87 avatar Nov 10 '19 09:11 zhangjinpeng87

Well, I'm not sure, I just do that for fun. Thanks for letting me know.

huachaohuang avatar Nov 10 '19 09:11 huachaohuang

How's the progress of this issue, are you still working on https://github.com/facebook/rocksdb/pull/5201? I'm interested on using this feature now.

huachaohuang avatar Feb 05 '20 09:02 huachaohuang

How's the progress of this issue, are you still working on facebook/rocksdb#5201? I'm interested on using this feature now.

I'm restarting the work.

yiwu-arbug avatar Feb 05 '20 18:02 yiwu-arbug