Idea: Content Addressable Storage abstraction

[jessekelly881]

What is the problem this feature would solve?

Continuing a conversation had on discord w/ @datner. https://discord.com/channels/795981131316985866/1251505447693979688

It would be nice to have a higher level data/blob storage abstraction that works across file systems that don't support the full node file system api (e.g. expo or tauri) as well as other storage providers (e.g. s3, google drive, etc.).

What is the feature you are proposing to solve the problem?

Adding a content addressable storage layer ObjectStore that can be created from not only a file system but also an s3 bucket, a cloud provider, etc.

export class ObjectStore extends Context.Tag("ObjectStore")<
  ObjectStore,
  {
    read: (hash: string) => Effect.Effect<Uint8Array, PlatformError>;
    write: (data: Uint8Array) => Effect.Effect<string, PlatformError>; // returns the hash
    exists: (hash: string) => Effect.Effect<boolean, PlatformError>;
    remove: (hash: string) => Effect.Effect<void, PlatformError>;
    list: Effect.Effect<ReadonlyArray<string>, PlatformError>; // list all hashes in the store
  }
>() {}

On a file system the file structure would look like this:

.dirname
    11966ab9c099f8fabefac54c08d5be2bd8c903af
    c90794e03b7da97d87016e50ac465d65d3a4f717
    ...

The actual implementations could be quite complex and could be combined allowing data to be logically split between multiple providers.

Previous art: https://www.npmjs.com/package/cacache https://git-annex.branchable.com/ https://github.com/max-mapper/abstract-blob-store

What alternatives have you considered?

No response