Portability of `list-keys`

[lann]

I think we should consider moving list-keys into another/a new interface. Its doc comment already lists a number of caveats which reflect the underlying issue that this feature just isn't very portable.

/// MAY show an out-of-date list of keys if there are concurrent writes to the store.

I think this comment even undersells the problem somewhat; some KV stores (memcached, at least) can't even guarantee that you will see a key that wasn't updated during iteration.

[Mossaka]

Thanks for raising this issue! I do want to revisit the list-keys API design, and as far as I know there isn't really a host implementation that uses the cursor field.

I know that list-keys is a common API for keyvalue stores and particularly useful for debugging purposes and for small datasets. Moving it to another interface sounds like a good idea to me.

wanna hear your thoughts on this @thomastaylor312

[thomastaylor312]

I do want to revisit the list-keys API design, and as far as I know there isn't really a host implementation that uses the cursor field.

We did implement this for redis and I think there might be a few others. I think most things can be implemented this way based on the investigation we did when adding that

I'm ok to move list-keys to another interface if it makes things easier for people. Should it be its own interface or should it be part of batch

[pchickey]

I think moving list-keys to batch seems appropriate unless someone has a reason it isn't.

I'd prefer to see the cursor string be replaced with a resource. That means the cursor is unforgable, so implementations don't have to concern themselves with what happens if the user invents or modifies the cursor, and it provides the implementation feedback, via the resource drop, of whether to pay costs because the cursor will be used to list more keys, or not.

Separate from moving to another interface, my proposal is, roughly:

resource store {
  list-keys: func() -> result<key-response, error>;
}
record key-response {
  keys: list<string>,
  cursor: option<key-response-cursor>
}

resource key-response-cursor {
  next: func() -> result<key-response, error>;
}

[AmitPr]

Just want to revive discussion on this with some comments:

1. list-keys, as is pointed out, is relatively narrow and without guarantees
2. For cases where list-keys does make sense, it might be more worthwhile to implement a more rich iteration / range API.

Specifically elaborating on (2), many embedded KV-stores (like RocksDB), allow for range iterations/reads. e.g. a range(low, hi, order) function. Thinking this could either be a part of batch, or (more likely) a iter/range world.

[dakom]

For cases where list-keys does make sense, it might be more worthwhile to implement a more rich iteration / range API.

agreed. As a low-hanging fruit, introducing a limit to list-keys() would be a good stopgap. The API as it is right now requires that the implementation decides the pagination size, rather than the component, which is unusual imho