Test data: TLS certs, proper management thereof

[philpennock]

We have 50+ TLS keys and certs inside the nats-server codebase, as test-data. Certificates expire and need to be reissued, and requirements change. We need a sane way to manage these test-data certs.

In test/configs/certs/ we now have both regenerate_top.sh and regenerate_rdns_svid.sh as shell scripts. These were created under time pressure, where the primary goal was "record exactly what is needed in each cert, and be able to recreate them all". There is duplication, there are missing features, and worst of all it's shell. But, it's there and it's an improvement over the prior state. Both of these scripts wrap OpenSSL; the top variant does so "a bit better", the rdns_svid variant is more low-level but was matching very exacting requirements.

We need to test various combinations, such as RSA keys, ECC keys, etc. With multiple relevant CAs.

This need is not specific to the nats-server; in nats-io/nats-top I previously wrote util/test/make-new-certs.sh which wraps github.com/cloudflare/cfssl.

Longer term, we need something better. I'm recording here my current thoughts on what we should probably have.

Because we need this in multiple nats-io repositories, it should probably be a new tool in a new repo, rather than ad-hoc inside the nats-server repo. (This repo would be the primary consumer, and was where we had the most recent fire-drill incident, so using an Issue here). We would then have a configuration file for that tool, in this repo.

Are there existing in-repo test-data CA management tools we could be using for this? The best tool is one we don't have to write, just configure.

Desired features:

1. Invoke within the repo and it finds all certs expiring within a time-window, default 2 months, and reissues them
2. Manages the CA private key being in-repo, or being nuked after each run; if the key persists, then when the CA is renewed, certs issued by it don't necessarily need to be. If the key is not persisted, then when the CA is updated then all certs issued by it need updating too.
3. Manage arbitrary sets of CAs; RSA, ECC, Ed25519 (future); manage "targets" on-disk.
4. Can recreate bundles as needed.
5. Each cert to be issued (including the CA) can have arbitrary attributes.
6. We can abstract out common parts of attributes, rather than repeat them a lot. Macros for common RDN prefices, for instance.
7. Minimally able to set CN, subjectAltName, EKU, etc. SAN support for DNS, IPv4, IPv6, and URIs (for SPIFFE).
   • Ideally much more flexible than just these limited options
   • Also, obviously, time-to-live
8. Minimize dependency hell.

I'm tentatively thinking of something like a Go tool with a TOML or YAML configuration file.

A v2 feature would be an associated GitHub Action, so that other repos can easily auto-generate a PR each month, updating certs which will expire within the next two months.

---

1. Does anyone know of a good tool which we should use for this?
2. Any feedback upon the features, or caveats to beware of?
3. Any additional fields we should be able to tune for our use-cases, from the beginning?

[philpennock]

Counter-Proposal: add test suite helper functions to manage CAs and Certs, and migrate from in-repo testdata to dynamically generated testdata. The keys/certs then never risk expiration.

[philpennock]

Added requirement: ability to explicitly mark a cert as deliberately expired, to test expired cert handling.

[philpennock]

Comparison tool: https://github.com/kairoaraujo/goca ... would need to edit a lot of tests to use the fixed file structure, but possibly worth while

Comparison tool: https://github.com/FiloSottile/mkcert -- is really geared for managing dev certs and local machine trust stores.