webcomponents.org icon indicating copy to clipboard operation
webcomponents.org copied to clipboard

[catalog-server] Ensure DB partitioning for tests and staging purposes

Open justinfagnani opened this issue 1 year ago • 3 comments

We want to be able to partition the Firestore database for running multiple tests agains the same database in parallel, and to potentially support multi-tenancy, different environments in the same GCP project, etc.

To do this we need to ensure that each collection has a unique name per environment. We do this now with a namespace suffix on the packages collection, but this isn't quite good enough because collection group queries could be performed on other collections. So we need to make sure that .collection() and .collectionGroup() are never called directly, but called through utility methods that add the namespace suffix.

We also need to update the design doc to describe this pattern.

justinfagnani avatar Oct 06 '22 01:10 justinfagnani

There's a problem with expressing namespace as collection suffixes: indices. Indices must be created via the console or CLI, and take a few minutes to setup even on empty databases. Indices are also configured by collection IDs, and are required for collection group queries.

The other approach we can take is to add a namespace field to every document and always filter on that field.

justinfagnani avatar Oct 06 '22 01:10 justinfagnani

And a problem with using a field is that we have a few places where we get a document by building a reference with known document IDs, and not by querying. We need collection suffixes to keep references unique.

So, we can do this:

  • For collections with synthetic document IDs, like customElements, we use a namespace document field
  • For collections with natural document IDs, like packages, we use a collection name suffix.

justinfagnani avatar Oct 06 '22 01:10 justinfagnani

This wasn't fully fixed yet

justinfagnani avatar Oct 15 '22 19:10 justinfagnani