Namespace config consistency

[ecordell]

From the Zanzibar paper:

Because changes to namespace configs can change the results of ACL evaluations, and therefore their correctness, Zanzibar chooses a single snapshot timestamp for config metadata when evaluating each client request. All aclservers in a cluster use that same timestamp for the same request, including for any subrequests that fan out from the original client request. Each server independently loads namespace configs from storage continuously as they change (§3.1.3). Therefore, each server in a cluster may have access to a different range of config timestamps due to restarts or network latency. Zanzibar must pick a timestamp that is available across all of them. To facilitate this, a monitoring job tracks the timestamp range available to every server and aggregates them, reporting a globally available range to every other server. On each incoming request the server picks a time from this range, ensuring that all servers can continue serving even if they are no longer able to read from the config storage.

SpiceDB currently caches namespace configs but does not enforce their evaluation at a particular snapshot. Without this in place, changes to namespace configuration needs to be carefully planned.

[ecordell]

Wrote a sketch with some thoughts in this doc

Instead of keeping global metrics on available snapshots, it could be simpler to just pass the namespaceconfig along during dispatch requests.

[jzelinskie]

@ecordell can we close this now that https://github.com/authzed/spicedb/pull/332 has been merged?

[ecordell]

We definitely evaluate queries against namespaces at a specific snapshot now, which I think addresses the main issue in the title. But there are some other aspects described here that we don't have covered yet:

1. We don't proactively update the namespace cache, we load it as needed (we don't have changelog for namespaces either, which is a prereq from 3.1.3 in the paper)
2. We don't pick a timestamp based on what nodes have available; instead we pick a quantized timestamp and let the nodes lazily load it into cache as the query propagates.
3. We haven't implemented the alternative to (2) that we've discussed, which is having the ingress node load the namespace config at the required revision and send it along with the dispatch requests, so that it doesn't matter what revisions the other nodes have loaded.

I think it would make sense to split these out and track separately - if we're not going to do them we should at least document why, and note that it's a deviation from the paper if we do. Personally I think 1 + 3 would be worth implementing.

[josephschorr]

With https://github.com/authzed/spicedb/pull/1681 now merged, the schema cache which supports updates in the background can now be used with Postgres, Spanner and CRDB

[ecordell]

After some discussion with @josephschorr, now that we have nodes watching for schema changes (point 1), I doubt we'll ever implement point 2 (or the alternative, point 3).

We almost always have the namespace cached now, regardless of revision selection. If it happens not to be cached, the node will query the datastore at dispatch time and cache it for all future requests, so being wrong sometimes is okay.

Currently, mysql doesn't have a namespace watch implementation, and we may need to introduce some sort of watch multiplexing for SpiceDB clusters with large numbers of nodes, but it seems unlikely that we'll start tracking the namespace highwatermarks and using them in revision selection: when is it better to pick a slightly older revision vs pro-actively loading the namespace if the watch is behind?