datenlord icon indicating copy to clipboard operation
datenlord copied to clipboard

feat: split current fuse function to fuse request layer and new storage layer.

Open IRONICBo opened this issue 1 year ago • 0 comments

Background

At present, the storage interface exposed to the outside world is POSIX (fuse), in order to consider the user space to complete the transmission of storage data, consider providing a set of sdk for the business layer to efficiently read the data stored in the back-end without going through the kernel, you can refer to the form of the deployment of the following two main forms of discussion. 1. a single datenlord service, multiple sdk processes to access the data through shared mem + (socket/pipe) to complete the data sharing:

  1. a single datenlord service, multiple sdk processes to access the data, through shared mem + (socket/pipe) to complete the data sharing, this time you need to provide a server to the service side. 2. datenlord as a lib, provide the necessary read and write interfaces to the sdk layer, other languages implement the corresponding c-bind to complete the interface access (prefer) The above two ways need us to extract the storage layer of read and write data interface, and provide HDFS-like read and write SDK, in order to ensure that the two access methods are not affected.

Core Modification

  1. Migrate memfs to src/fs and rename to datenlordfs.
  2. Split memfs to FuseFilesSystem and VirtualFs.
  3. Move fuse params to fs_utils.
  4. Remove metadata fuse operation.

TODO

  1. buffer trait, current implmentation is use BytesMut instead,we need to use common trait.
  2. modify direntry to buffer

IRONICBo avatar Aug 29 '24 06:08 IRONICBo