Access raw certs provisioned via ACME

[mikehardenize]

I don't know if the following is an enhancement request, or simply a request for info on how to do something. I couldn't find a more appropriate support channel, irc/gitter/forum etc.

What would you like to be added

I would like real-time access to the raw certificates that are provisioned by the ACME provisoner. I'm happy to just tail the logs to get this information.

Why this is needed

So I can feed these certs into our private CT log as they are generated.

[dopey]

Hey @mikehardenize, thanks for opening the issue! We actually have a stale CT log support branch that we never got around to merging. Don't have an ETA yet, but it's in our short term backlog to dust that branch off and get it merged.

[mikehardenize]

Thanks, that's good to know. In the mean time, what's the best/simplest way for me to pull certs out of step-ca? I can push them into CT once I have them, pretty easily.

[dopey]

Unfortunately, the only way to do this right now is to query the database.

If you're using badger, you'll need to shutdown the CA and run a script against the db to dump the certs. If you're using mysql, you can keep the CA running, but you'll still need to run a script.

The data, in both mysql and badger, is stored in a nosql pattern - key, value byte blobs. So you can't query it without a script. I have a script I used with badger to dump the number of entries in each table (for some private debugging I was doint), that you could use as a "kick-off" point in getting your own script working. Happy to link you to it if you're interested. For mysql, the script would look more or less the same.

Going forwards, we have this open issue (https://github.com/smallstep/certificates/issues/239) that we plan to address in the short term. Additionally, we plan to switch the default DB to a sql DB using sql storage, so you could just query it as you would a normal sql DB.

[anthonyjlmorel]

Hi @dopey ,

Is there any further information about the format in the blob entries? I am trying to request a mysql DB to get information about issued and revoked certificates, using NodeJS. I am able to read the nvalue entry for the revoked_x509_certs table, as this is a basic JSON object stored into the nvalue entry, by doing:

const entry = /* ... get entry from DB */

const entryValue = JSON.parse( Buffer.from( entry.nvalue, 'binary').toString() );

// can use the object 
console.log(entryValue.Serial)
console.log(entryValue.ProvisionerID)
console.log(entryValue.RevokedAt)
// ... etc.

But when accessing the x509_certs table, I cannot figure out how can I "decode" the BLOB, even with the gist you gave. Do you have more insightful example for this table?

Thanks !

[dopey]

Hey, @anthonyjlmorel the best documentation/info I can provide (other than actual documentation - which we don't have ;p) is to link you to the places in the code where we do the querying / transformations.

https://github.com/smallstep/nosql/blob/master/mysql/mysql.go

I would recommend using the golang nosql methods - only because I know for sure that they work.

Based on your code snippet, once you get the binary data from the db for a particular key. You'll need to "unmarshal" the data into the type used by the revoked_x509_certs_table -- here https://github.com/smallstep/certificates/blob/master/db/db.go#L97-L107.

Just below that there is an example of writing that data to the DB -- https://github.com/smallstep/certificates/blob/master/db/db.go#L155-L171. We just marshal the struct into binary and key by the serial.

Let me know if this helps to unblock you.

[anthonyjlmorel]

Thanks @dopey , will check this out !

[dopey]

Hey, circling back on this issue - we've released certificate observability (visibility, alerting, monitoring) as part of our hosted product. You can "link" an open source CA to the hosted product and start seeing new certificates in the UI.

Linking a single open source CA is free. Sign up here: https://info.smallstep.com/certificate-manager-early-access-mvp/, and follow the directions to link an existing CA. Encourage anyone looking to get visibility into their step-ca PKI to try this out and give us feedback. cheers!

Note: you'll need to recreate your provisioners using the new step beta ca provisioner subcommand group. Let us know if you're having any issues there and we'll be happy to help. For real time help, come find us on Discord - https://discord.gg/ypu2T7qg9y.