Root signing key rotation and prepublishing

[schmichael]

Nomad 1.7 uses a root encryption key to encrypt Variables at rest and a root signing key to sign Workload Identities.

These root keys should be rotated automatically using the following logic:

1. New root key should be generated at root_key_rotation_threshold / 2 and the public signing key published before use in the JWKS endpoint.
2. At root_key_rotation_threshold the prepublished key will be made active and the old active key will be made inactive
3. After root_key_rotation_threshold + root_key_gc_threshold after the old key was marked inactive, it should be garbage collected.
   • Update root_key_* docs to reflect that keys are not gc'd until rotation_threshold + gc_threshold are reached to avoid invalidating otherwise valid JWTs in use.
4. Jobspecs with an identity.tll > root_key_rotation_threshold should receive a Warning on submit.

Prior Art

https://github.com/hashicorp/vault/pull/12414

[tgross]

What you're envisioning here should cover https://github.com/hashicorp/nomad/issues/19367 and https://github.com/hashicorp/nomad/issues/19368. I'm going to unassign myself from those and if you want, we can either close them out now or you can close them out with this issue.

[schmichael]

Thanks for linking things together Tim.

we can either close them out now or you can close them out with this issue.

I'm going to leave them open until this ships to ensure everything is buttoned up appropriately.

[tgross]

Implemented in #23577 and will ship in the next regular release of Nomad 1.8.x, with backports to Nomad 1.7.x/1.6.x Enterprise.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.