matrixone
matrixone copied to clipboard
matrixorigin
[Enhancement]: CN read cache
Is there an existing issue for enhancement?
- [X] I have checked the existing issues.
What would you like to be added ?
Design CN cache, row or column format. S3 cache.
Simple impl based on S3 file immunity.
Work out with Storage team on what does this mean for inserted data.
What about update/del chain.
Why is this needed ?
No response
Additional information
No response
CN read cache should get a memory budget and ssd budget on startup.
- The IO unit, as we is one vector (col group), target 8K.
- Zonemap reading should also be cached.
- The cached content does not need to be the same as on disk/s3 format, for example, if a vector is compressed, maybe we should cache the decompressed content.
- Either a Pin/UnPin interface, or, an agreement with query executor, that executor should get vector from cache and operate on the vector. There should be NO memory allocation/mem copy to use vector from cache.
https://github.com/reusee/matrixone/commit/26371a571ae8f97fb2581aacd86b8ee800aec14f I made a proposal for further discussion
This is fully implemented. Related changes: https://github.com/matrixorigin/matrixone/commit/b2eefef95b79ddfd17e926ead185a312750f80d9 https://github.com/matrixorigin/matrixone/commit/670136a87fe71a519d748e0a5e31132061036be1 https://github.com/matrixorigin/matrixone/commit/730de8bee3a4e0277fce4d761fd53d4ef1670117 https://github.com/matrixorigin/matrixone/commit/3ffbdf3a171b7819a7db65bf21bfd692689fc47a https://github.com/matrixorigin/matrixone/commit/0d89c5d454944802073f1a338965e16ccd493845 https://github.com/matrixorigin/matrixone/commit/610c908fea6ab125ca1c97fa650a414e6b616aea
@reusee Now CN cache takes another new meaning and much more complex. Do you want to reuse this issue, or have another issue tracking it?
@fengttt I think this one includes the remaining CN works: https://github.com/matrixorigin/matrixone/issues/3931