corestore

corestore copied to clipboard

What do you think about adding a { persist: false } option to corestore.get? It would create an in-memory hypercore that behaves just the same as stored hypercores, with regard to both key derivation (from the master key and namespaces) and replication.

The dat-sdk currently has some code to "patch in" in-memory cores into corestore's replication logic, using internal methods. This could be removed then. There's several usecases for replicated in-memory cores I think.

Can you describe the use case for this a bit more? Makes things inside the corestore much more complex in terms of lifecycle when storage isn't consistent

I use it in applications where I want to load data temporarily and throw it out after I'm done while using the same corestore to manage persistent data.

Flow could be something like:

• Create persistent hyperdrive
• Load a core in-memory to fetch a bit of data from it
• Place data in hyperdrive
• Close core because I don't need it anymore

You can use hypercore(ram) for that with no added complexity currently. Not sure what’s gained by adding that to the corestore.

If you use hypercore(ram) you need to set up the swarming for it separately from the stuff you're already doing for the corestore. 😁 It can be nice to leverage connection reuse / reduce overall network traffic.

Adding this to corestore would definitely simplify getting the temporary cores hooked up to networking, but as @mafintosh said it'd come with a complexity cost. I think the need for this motivates one or more of four things:

1. A corestore method that temporarily inserts an externally-managed core into a corestore's caches + replication streams.
2. A userland module that wraps a single-corestore networking module (like corestore-swarm-networking) and makes it easy to push other cores into its replication streams.
3. A corestore deletion API (which I know you're keen on @RangerMauve, but which we've been hesitant to do until a good approach to key backups is available).
4. A corestore persist: false option.

The persist: false option seems the least desirable because of its complexity -- the corestore API surface/internals should be as small/simple as possible, and we're already pushing it IMO!

I'd personally opt for (2) and/or (3). (2) because it's userland which is always nicer, and (3) because it's something we need to tackle anyway, so we'd get many benefits elsewhere.

For what it's worth (3) could suit the needs of temporary cores, and it doesn't need any special APIs inside corestore. You could do feed.destroyStorage() which'd do the trick. 🤷 I think it'd require developers to be aware that they should clean up cores they don't want stored anymore.