PseudoKV

[MeijeSibbel]

Moving this here-->The metafile format was designed to store ordinary user files, and PseudoFS was designed to represent an ordinary filesystem. Neither of them will work optimally with an S3 bucket architecture. It sounds like we should investigate creating something like a "PseudoS3" that is better geared towards key-value storage.

[lukechampine]

To make this slightly more concrete, I was thinking along the lines of:

type PseudoS3 struct { ... }

func (PseudoS3) CreateBucket(name string) (*PseudoBucket, error)
func (PseudoS3) Bucket(name string) (*PseudoBucket, error)

type PseudoBucket struct { ... }

func (PseudoBucket) Put(key, val []byte) error
func (PseudoBucket) Get(key []byte) ([]byte, error)
func (PseudoBucket) Delete(key []byte) error
func (PseudoBucket) Sync() error

The main goal would be to aggregate Puts across the entire PseudoS3 instance, such that data is only flushed to hosts when a full sector's-worth of key-vals have been stored. However, I'm not familiar enough with S3 to flesh out a full API, and I don't have a good idea of what the local storage aspect would look like.

[jkawamoto]

I think it also needs List function, too.

[jkawamoto]

I think it'd be better to call this interface PseudoKVS instead of PseudoS3 because S3 API has more functionalities such as versioning and ACL. Using S3 sounds confusing.

[jkawamoto]

It also would be better if the type of val is io.Reader in both Put and Get instead of []byte. If it uses byte slices, we need to copy each object to heap, and thus we might not be able to store bigger objects.

[MeijeSibbel]

PseudoKV has been integrated, moving new roadmap features to https://github.com/lukechampine/us/issues/126 for consistency.

[jkawamoto]

I think PseudoKV isn't done yet.

There are still some missing functions:

https://github.com/lukechampine/us/blob/8a2eb347750d906fe1a10811caef12271df28073/renter/renterutil/metadb.go#L348-L354

https://github.com/lukechampine/us/blob/8a2eb347750d906fe1a10811caef12271df28073/renter/renterutil/metadb.go#L367-L369

Also kv branch is not merged into master.

[MeijeSibbel]

Thanks, reopened. I see this is also related to https://github.com/lukechampine/us/issues/102 correct?

[jkawamoto]

Yes, it is. Especially, this problem https://github.com/lukechampine/us/issues/102#issuecomment-689943901.