Implement Storage Engine

[skyzh]

Development Task

Implement Storage Engine, R/W API and Transaction API (3-4 weeks)

Development progress

This section tracks what has been done (on develop branch)

• [x] Implement LSM tree and compaction
  • [x] memtable
  • [x] L0 table
  • [x] multi-levels with simple compaction policy
  • [x] compaction to base
  • [x] persistent database (currently it's just write-only)
    • [x] delete unused files
    • [x] manifest
    • [x] persist files and recover
  • [x] compaction
• [x] Implement value log
• [x] Implement database iterator
• [x] Implement MVCC and Transaction API
• [x] Patches and Fixes
  • [x] implement merge iterator
  • [x] implement concat iterator
  • [x] fix varint

Pull Requests

This section tracks progress on master branch

• [x] memtable https://github.com/tikv/agatedb/pull/28
  • [x] first, refactor current structures https://github.com/tikv/agatedb/pull/37
  • [x] WAL https://github.com/tikv/agatedb/pull/55
  • [x] vlog https://github.com/tikv/agatedb/pull/82
• [x] varint https://github.com/tikv/agatedb/pull/29
• [x] merge iterator https://github.com/tikv/agatedb/pull/30
• [x] concat iterator https://github.com/tikv/agatedb/pull/65
• [x] extra tests https://github.com/tikv/agatedb/pull/38

[zhangjinpeng87]

@Fullstop000 Do you have an interest to participate in this work?

[Fullstop000]

@Fullstop000 Do you have an interest to participate in this work?

Sure :)

[g302ge]

@Fullstop000 Do you have an interest to participate in this work?

mee too :)

[skyzh]

@Fullstop000 Do you have an interest to participate in this work?

mee too :)

I will post a future roadmap soon, and we may discuss what to do next later :)

[skyzh]

@Fullstop000 Do you have an interest to participate in this work?

mee too :)

I think the critical missing pieces are:

• vLog GC
• block + index cache and compression

And there are a lot of small things for improvement, for example:

• use thread pool for SST merging
• reduce vLog lock contention
• and so on...

And finally a big design decision for the future of AgateDB:

• Do we really need to implement MVCC capabilities of Badger? It seems that a KV store with the snapshot capability is enough for TiKV.

After all, there is no clear development plan for AgateDB for now. If you have any idea or question, feel free to ping @skyzh on TiKV Slack. cc @LeeMaster