spicedb icon indicating copy to clipboard operation
spicedb copied to clipboard

Improvement: Prevent ZedToken's from being used cross-datastore

Open josephschorr opened this issue 2 years ago • 2 comments

Right now, ZedTokens simply encode the datastore-provided revision information, with no additional information as to the kind of datastore used. This can result in weird behavior if a ZedToken from one SpiceDB cluster is used with a completely different cluster, or the datastore has been swapped out.

We should instead encode the datastore kind (and perhaps a unique ID) into the ZedToken, and either reject all such tokens, or automatically switch to full consistency (likely the former)

josephschorr avatar Sep 19 '23 19:09 josephschorr

Please consider that stored tokens can be scattered across multiple services managed by multiple teams. Somehow invalidating all of those tokens and keeping new ones from saving during a datastore change can be a herculean effort. I'd like to suggest that SpiceDB's response to tokens that don't match its current datastore be configurable.

To wit. If I were to have this kind of control and have to migrate data stores, I'd perform the following steps:

  1. Configure SpiceDB to switch to full consistency when it sees a legacy token.
  2. Cut over data stores.
  3. Wait thirty seconds for the SpiceDB caches to become fresher than extant pre-migration tokens.
  4. Reconfigure SpiceDB to instead consider all legacy tokens to be far enough in the past that the current cache should be used instead.

I may opt for intentional downtime and instead do:

  1. Start my maintenance window and shut off access.
  2. Configure SpiceDB to consider all legacy tokens to be far enough in the past that the current cache should be used instead.
  3. Cut over data stores.
  4. End my maintenance window.

MichaelCordingley avatar Oct 25 '23 16:10 MichaelCordingley

I think making it configurable is reasonable

josephschorr avatar Oct 25 '23 17:10 josephschorr