spicedb
spicedb copied to clipboard
Improvement: Prevent ZedToken's from being used cross-datastore
Right now, ZedTokens simply encode the datastore-provided revision information, with no additional information as to the kind of datastore used. This can result in weird behavior if a ZedToken from one SpiceDB cluster is used with a completely different cluster, or the datastore has been swapped out.
We should instead encode the datastore kind (and perhaps a unique ID) into the ZedToken, and either reject all such tokens, or automatically switch to full consistency (likely the former)
Please consider that stored tokens can be scattered across multiple services managed by multiple teams. Somehow invalidating all of those tokens and keeping new ones from saving during a datastore change can be a herculean effort. I'd like to suggest that SpiceDB's response to tokens that don't match its current datastore be configurable.
To wit. If I were to have this kind of control and have to migrate data stores, I'd perform the following steps:
- Configure SpiceDB to switch to full consistency when it sees a legacy token.
- Cut over data stores.
- Wait thirty seconds for the SpiceDB caches to become fresher than extant pre-migration tokens.
- Reconfigure SpiceDB to instead consider all legacy tokens to be far enough in the past that the current cache should be used instead.
I may opt for intentional downtime and instead do:
- Start my maintenance window and shut off access.
- Configure SpiceDB to consider all legacy tokens to be far enough in the past that the current cache should be used instead.
- Cut over data stores.
- End my maintenance window.
I think making it configurable is reasonable