KV - Implementation details

[missinglink]

Heya, Not so much an issue as kicking off a discussion ;)

I'm curious about the feasibility of implementing higher-level abstractions (such as Set, Map, Bitmask, etc) on top of the Cloudflare KV API (much like the node community did with leveldb).

Leveldb itself doesn't really offer many 'core' APIs, but it does crucially support deterministic sorted key listing and prefix searches, something it looks like KV supports, or partially supports?

Here's an issue I opened over at the Cloudflare forums to get some clarification. https://community.cloudflare.com/t/kv-implementation-details/107268

Thunderclap looks great, from what I could see in the source, you're paging through all the results from a keys() operation and then presumably you'd need to do the sorting in the application (rather than have the DB do it for you)?

[anywhichway]

@missinglink If you look in the file src/cache.js, which is the code that actually calls the Cloudflare APIs you will see the method 'keys'. This does take a prefix and passes it to Cloudflare. Your questions to Cloudflare are good ones. My experience has been and the entire Thunderclap architecture depends on these being deterministically and lexigraphically sorted. That being said, I have asked about consistency and architecture myself and not got much in the way of useful responses. It would be nice if they published a white paper on their architecture.

I have noted some pretty substantial lag on value retrieval and updates with the Cloudflare KV. I am working on an lmdb backed version of Thunderclap that runs on a centralized server 1TB SSD drives. Current tests seem to indicate that for domestic uses this will be much faster and the architecture of the underlying store is well understood.