Cherry-pick TiKV related changes to 8.10.fb

Open v01dstar opened this issue 1 year ago • 0 comments

6.29 (last TiKV base) diff: https://github.com/facebook/rocksdb/compare/6.29.fb...tikv:rocksdb:6.29.tikv

Apply write-amplification-based rate limiter fe76269d5cc395e37e746a3e9990dd38ed6ec5b0
- 3dc1b55d5ae28a4872dcbcb64c8731fec4c0e392
- 458bbd7f3d0ebc1887e05044c65e86fd9bebb6df
- ccf52153aee2f12df5a05cfd7c354fcfc9de202f
- f9aacb309221f027f90153c0524cfb6b1963eded
- 4751586b756fc449ea9ffd090844f4bfdff38655
- 8d6414ef21432e5f330c504891e6a85784459bcb
- db59a3f7aeb142823b410ad8e7beacaf3bf259af
- c0953b3c0ae24b9764149b3c364d1650c5850781
- 938c016175e84406888dfbf30c93844e9b124a3b
- Comments:
  - Env::IOPriority has more options now, LOW, MID, HIGH, USER instead of just LOW and HIGH, write amplification rate limiter's tuning logic may need to adjust to this change.
Apply PerfFlag patch b27b56482830ace37d4f7ce813e9f4ddffead952
- 2f1efc5364b9106f46e7333c6987144327d28a50
- 7ee5329eeac4dede9664b1b7244b8fc325eab805
- Comments:
  - We need to resolve compatibility issue every time when RocksDB adds new metrics. And all dependent projects, like Titan and rust-rocksdb have to change accordingly.
Compaction filter optimization 2a936871a09e1f45b9fa26d8f20403b87c5c5a6f <-re-evaluate unsafe filter v4
- 8e203492b81af0df1cbda907d5f0e8fbaf13d266
- 9554ad2b3fa7de8928054bfb30c2fcd6e714c50e
- ~~23c8635d6de07f9984ebfbb076be0c5b2f395bd3~~ <- removed (this is a bug)
- 3a238bf6f367b5ee76de9c497eba823c0e156ead <- no longer needed, see https://github.com/tikv/tikv/pull/9694
- Manual apply related changes in bb515db377c3bc71eddbe89a46d0137cffbbcbf4
- Comments:
  - We also need to resolve compatibility issue every time for this when RocksDB changes filter API.
Doubly skiplist for reverse scan acc80ceebcb0a89d738e7a00e1ea2681324af64d
- 5d01038a5d25a0acc6fbeb24d8418cb0cfc4a9b9
Add WAL write duration metric de0fc3091719096a073ad158b66fb649216ef3a9
- e3c8f48cb425c1211a298a9875609f14664dfa95
Add Iterator and Append method for WriteBatch 6ffdf203ac8bba76015672898546ea82f9d4b8a5
- 340b810338ed75a9e913c1ab9e3d80df41d12cf5
TiKV IO rate limiter a8d22fcfd32421b547419ba23262417e6b90a7f9
- 774986015c6e5c7fbc01e8b0b0c80f66c4b18a42
- 52b4a97064a4568a34d00d95affd361e60ef52cf <- retiring because of https://github.com/facebook/rocksdb/pull/9311
- 4ec4a1fafb9212db4377fee5b114c8e18d2c837a
- 8bd42cac8395331e78ae705eebdacc767c7d4f26 <- removed base_background_compactions related code due to https://github.com/facebook/rocksdb/pull/9462
- 28f7636c315e9276a15f7e55feae0ba9c0fca3c7
- Remove ROCKSDB_LITE macro use
Manifest dump tool optimization
- e1797a06ab03560622b8884a8d8f873a0dc6fbe8
- 615c1dc21fa61ce077a01fb313689c74446bb9e3
Implement pipelined commit / multi-batch write
- 910417bc69bb152f687f100c6d25216796c059d1
- 19db40b25f400343acd1a99468f24d4033a743e8
- 2d03d536ffe7b80628e694010526f801f9a949d3 <- removed, but we need to verify this does not cause upgrade issue again tikv/tikv#13007
- f4cba2fb5ac51d59d076c3626fc111b0c0cad5bc
- 9bb7147bf05284e96abebb7de3ff8faff63aa582
- Manually applied some changes from 98a80e979bf1554823b0340dc2873633069ec16a
- Comments:
  - This is not easy to maintain, multi-batch write's implementation is based on rocksdb::BatchWrite() (with some custom code). Whenever rocksdb::BatchWrite changes, multi-batch write's implementation should change accordingly, this requires human intervention.
Per-file encryption key management
- b9c2064666e3d2b7960f4f6bfc383b3d7b7061b6
- 63869927da3db0de304831c5a3af8b96eb65d1d1
- 63399dfff11699b5c0df754915e5b2f44c4ae664
- 63586f274036c6127c033c2b0b61fd5e6596baaa
- 7ee32c046ae9d57857afd1b8475048eea4550a37
- bbd27cf817310f015e415339bbec15fd1ff7e3c6
- 1868d12ec29f1d27670474120893d4e62613473c
- 4cebfc19e75475b9aa7937da46e9c8fc28c5d738
- 946476657ffd4ffadad754d36e2c3b22687ce209
- Remove ROCKSDB_LITE macro
Optimize SST partitioner to avoid huge compaction
- e2f6ec7e27195082a336180e9f68448d1a8dbc5c
Titan
- Make statistics extensible 6d88b39dba1c14f7ec444c1b3b489b0c72e90887
  - 7c6dcaa496564ec49cb1a135cfbf9e0a22cb31c5 <- pitfall, do not try to move statistics impl to stats.cc. https://stackoverflow.com/questions/1111440/undefined-reference-error-for-template-method
  - Manual apply changes in bb515db377c3bc71eddbe89a46d0137cffbbcbf4
- Manual apply related changes (blob index) in bb515db377c3bc71eddbe89a46d0137cffbbcbf4
- dcf2f8d56092285381be2acbf8f04b8aeeb7ad79
Raftstore v2?
- 32f8f2b45146904ce1b6c44cad8fd49086d67547
- faad4834fff2aef36967471cf5c9e54fa0efc361
- 8899a360623988adeec1d865bef9d24568ddd32c <- remove wal
- 638c217b61638985ebf56a3389c105c6b6e4c11d
- Expose seqno / Add post write callback
  - 3cd757c0e829472f53a9ff9b3a09359c2e5ee522 <- over-written by 08aa50323af00ae4e4681aab34973172e81045ad
  - 08aa50323af00ae4e4681aab34973172e81045ad
  - fdcd14dbbea3abdaac8e89cf9c95f6094b914424
- 9ea79ab44ac0d597d8609e8ffdf4360dae2f1a94
- acc624ff90e3cbaac73efe8a31635d2ce23345ff
- cd9aa9993fda09083e8ac63d567cb71c0fc58844
- 8a9c10ead01a564fc1d5b1a4911163aa72e3c7df
- 14f36f8ccb7973a2a675fc54b02d130021563b97
- 5b9cef986931a129bc6009391df768ebf1a7af6b <-re-evaluate
- de47e8ece9abf001af74a9e60f66b9c8494240e1
- 6121b2dcc8eb8e0e579178d949e94f9d0691c2f5
- 0813e37a400bd058d6d160c01475af960626a65a
- fe76937087f81e95a323bbf00fcd109c7b39c649 just the CheckInRange API

Already exist in upstream (fb):

d33a7c70e60c69b3545cbe42b3d0f070fc768b7a
2ac42c6511d12f4f4288ff9a0ba988376d40fd1a
310c8185cb6db6d115949efc1dc233315068e035
61809fe5d6c35d47b9991e5e2e245348acc62ccb
1726df0dbcacbc28ae012c228292f31567e13bb1
9c86bf58527d032b2ec4dca18466f59e26a2ae8e
ee9729130ea1a8d0d451ed8866889934cb456dfb
8b1ab20b698d02adf103f9abbb4e7ad7c2e0873d
8c7ff07bb2eec4aa650a2acb09f2bfd828f1d033
267baedd39546c68cb9b554cad47c2225a537ecc
d453c8e1eb7066a9b965085a8ae37609becafdb8
e33a23322ba7c00849c1c0056dbf90f7c57412b2
22304c4579f4b6bdce14f83a4f7fef6ff7d76828
1022df7002f994aac90750edafb396df4c24035e
98a80e979bf1554823b0340dc2873633069ec16a
6a7472ae4fd0699c2d533b03f153bfddf5f76032
ce0ea61aecd53ac02a0e7da2b166961c7d98650a
4d12477c12aaf193de9e00282a1403f5b3421741
45509f0f530ad370863876fc1ee95ccf85bfe96d
20225a41c5f3c9d9c3d44bf50a1ac5995ccfcb67
3dba9fa2067a2f979856e255890ab9783e86db98

No longer needed:

bc1f25519feeef8387df869f61b8aa79fa2f1de3
dc9353fc7ce5415cf3180bcb1af2cfdad964d7ca <- fixed differently in upstream https://github.com/facebook/rocksdb/issues/5892
545d0b22635322f8ba2f6d37ec55ea8b634efacd <- fixed by https://github.com/facebook/rocksdb/pull/10922
bb515db377c3bc71eddbe89a46d0137cffbbcbf4 <- Separated. i.e. monitoring related changes are merged with "Make statistics extensible"
2cbb06984022c417adeba5f82a3c0b143f4f93db <- reverted by 53eae8296dd47321f968ea59f0bfb9d4ecd7878e
53eae8296dd47321f968ea59f0bfb9d4ecd7878e <- reverting 2cbb06984022c417adeba5f82a3c0b143f4f93db

Need triage:

0559eac70cc21cc51222df947ba182986cd2bf1b <- rocksdb cloud

To be verified:

40551e21c637c88393cceb5fc662c72ff158545e <- we can evaluate RocksDB's new option for solving this problem introduced in https://github.com/facebook/rocksdb/pull/11701

Complications:

WriteBufferManager has changed a lot
SST file epoch number was introduced, instance merge needs to accommodate that change.
Write stall logic behavior change introduced in multi-instance support project made some tests to fail.
RocksDB now uses C++17 standard

Apr 11 '24 17:04 v01dstar