Reduce inaccurate 404s by adding structure to ACME object URLs
Now that we're doing more with sending traffic to replicas, we have a problem where sometimes a user creates an object (account, order, authzs), and then immediately fetches that object. If they hit a replica for the fetch, and the replica is lagged, they might get a 404.
At a minimum we'd like to turn these 404s into something more informative that indicates a retry might succeed (like a 5xx series error). We might also like to route such prospective 404s to the primary DB. But we don't want to send all 404s to the primary DB because that would be too much junk traffic.
We can start incorporating creation timestamp into the URLs of objects we create. This will allow us at request time to determine that an object was recently created, and query the primary DB if the query to the replica returns no answer.
To ensure that the timestamp is meaningful and not random junk, we can add an HMAC (ht @mcpherrinm).
Adding structure to ids will also be useful as we start to horizontally shard. It will be useful to know which shard an object was created in, or which datacenter.