Manage service account keys between shards

[kylape]

There needs to be a way to share service account keys between shards.

From @sttts:

We might want to start thinking about a multi-shard deployment. Then we likely have to put the service account keys into a secret and share that secret between shards. We might also want to have service account key per shard, but distribute trust (some CA? Does that work with service accounts? cc @marun).

Additional context PR comment

[marun]

Token signing keys are rsa, so each server would either share the same key or have an independent key. There is no option for a certificate-based approach with CA afaik.

[csams]

@marun @sttts If all shards have independent private keys and all shards should accept SA tokens from all others, we'd need to distribute the public key portions of the private keys for signature validation. We have --service-account-key-file in the authentication options, so I guess we could come up with a way to populate the internal list behind it as shards come and go?

[mjudeikis]

@csams , do you remember, would it be enough if SA keys share trust for this to work or do we need something else?