Support for CRL

[unreality]

Description

This pull adds a new API endpoint /crl that will return an on-demand generated PEM-encoded CRL. Fixes #206

The CRL is generated on-demand:

• if it is requested after the NextUpdate time specified in any previously generated CRL
• if a certificate has been revoked
• if no CRL has been generated previously

A new database table for storing the CRL named x509_crl is created when step-ca is launched.

A new interface CertificateAuthorityCRLGenerator is available for CAS object to implement. If they implement the interface, CRL lists will be created, otherwise a 'not implemented' error is raised when the request for CRL is made.

[CLAassistant]

All committers have signed the CLA.

[unreality]

I changed GenerateCertificateRevocationList to return a DER formatted list, it seems many clients prefer to receive DER instead of PEM, so this should make it easier to support that.

[maraino]

@unreality You're right about the DER format, if you look at the CRL distributions points embedded in certificates, they are DER encoded.

I will talk about this PR with the team in our triage meeting, I have some ideas that want to bring to the team before moving forward.

[maraino]

@unreality I need to look at this again, but can you do some experiments and see how the enumeration of revokes performs with millions of revokes?

[unreality]

I'll see if i can test the speed of the crl gen based on a large number of revoked certs, but no promises :)

I'll also try fix all the linter errors im seeing.

Another issue ive come across is that when attempting to verify the CRL under windows, the mechanism complains about needing a client certificate to get the crl:

certutil -URL https://yourca.example.com/crl

Returns:

Error retrieving URL: A certificate is required to complete client authentication 0x80072f0c (WinHttp: 12044 ERROR_WINHTTP_CLIENT_AUTH_CERT_NEEDED)

I'm assuming this is because step-ca prompts for mTLS on any api endpoint? Is there any way of excluding a route from prompting for mTLS?

[maraino]

I'm assuming this is because step-ca prompts for mTLS on any API endpoint? Is there any way of excluding a route from prompting for mTLS?

step-ca only requires a client certificate (mTLS) in the /renew endpoint. If you send it to any other endpoint it will be verified, but it's not required. We basically use Go's tls.VerifyClientCertIfGiven and then on the methods that require mTLS we check if a certificate was given.

My guess is that that certutil might have a bug in how it manages client certificates, but you can always create a client certificate. By default all certificates signed by step-ca can be used for client authentication. This is the base on how or /renew endpoint works.

[dopey]

Hey @unreality let us know if you have any questions about the feedback. Cheers 🍻 and Happy Holidays!

[unreality]

Apologies for the delay on this one, now im back at work i hope i'll be able to get this cleaned up for merging soon

[LarsBingBong]

Will this also make OCSP work? Me asking in > https://discord.com/channels/837031272227930163/841243933936975912/939177692874555502

[maraino]

@LarsBingBong an OSCP responder can be created combining a couple of things, see details https://gist.github.com/maraino/2905ba742168d04ddca9882ac1f0c3ed and here for badger https://gist.github.com/maraino/4dcb64cb051b17ef6d421892cb4e55a8

[unreality]

I've pushed most of the requested changes, let me know if there are still issues that need resolving :)

[LarsBingBong]

Exciting @unreality - thank you - we can really use this for our Smallstep CA setup - if @maraino can accept.

A great day to you both.

[LarsBingBong]

@LarsBingBong an OSCP responder can be created combining a couple of things, see details https://gist.github.com/maraino/2905ba742168d04ddca9882ac1f0c3ed and here for badger https://gist.github.com/maraino/4dcb64cb051b17ef6d421892cb4e55a8

@maraino I can't see how I can use this together with the Helm chart to the step-ca. There's no go client/cli installed. So it can't be executed inside the step-ca container on Kubernetes. And the PVC is RWO so it can't be mounted to multiple POD's.

[maraino]

@LarsBingBong an OSCP responder can be created combining a couple of things, see details https://gist.github.com/maraino/2905ba742168d04ddca9882ac1f0c3ed and here for badger https://gist.github.com/maraino/4dcb64cb051b17ef6d421892cb4e55a8

@maraino I can't see how I can use this together with the Helm chart to the step-ca. There's no go client/cli installed. So it can't be executed inside the step-ca container on Kubernetes. And the PVC is RWO so it can't be mounted to multiple POD's.

Those scripts are not supported at all, badger only supports one reader at the time, it might work if you copy the database files to a new directory, but you will also need a new image with the required scripts and the binary version of that .go.

And with this PR merged, you will be also able to create an OCSP responder using the generated CRL as the source.

[unreality]

Committed some additional changes, but i'll need to merge or rebase to continue i think, so i'll do that when I have some time

[LarsBingBong]

@LarsBingBong an OSCP responder can be created combining a couple of things, see details https://gist.github.com/maraino/2905ba742168d04ddca9882ac1f0c3ed and here for badger https://gist.github.com/maraino/4dcb64cb051b17ef6d421892cb4e55a8

@maraino I can't see how I can use this together with the Helm chart to the step-ca. There's no go client/cli installed. So it can't be executed inside the step-ca container on Kubernetes. And the PVC is RWO so it can't be mounted to multiple POD's.

Those scripts are not supported at all, badger only supports one reader at the time, it might work if you copy the database files to a new directory, but you will also need a new image with the required scripts and the binary version of that .go.

And with this PR merged, you will be also able to create an OCSP responder using the generated CRL as the source.

Yeah that was my conclusion as well. Fully confirmed with your comments. Much appreciated. Thanks! Looking forward to this merging at some point - in hopefully the near future.

[logopk]

@LarsBingBong an OSCP responder can be created combining a couple of things, see details https://gist.github.com/maraino/2905ba742168d04ddca9882ac1f0c3ed and here for badger https://gist.github.com/maraino/4dcb64cb051b17ef6d421892cb4e55a8

@maraino I can't see how I can use this together with the Helm chart to the step-ca. There's no go client/cli installed. So it can't be executed inside the step-ca container on Kubernetes. And the PVC is RWO so it can't be mounted to multiple POD's.

Those scripts are not supported at all, badger only supports one reader at the time, it might work if you copy the database files to a new directory, but you will also need a new image with the required scripts and the binary version of that .go.

And with this PR merged, you will be also able to create an OCSP responder using the generated CRL as the source.

@maraino could you please explain how this can be done?

Currently I'm using openssl ocsp, that needs the index.txt to serve the ocsp responses. That's why I created the listcerts.go for badger to create the plaintext file.

Do you know of any good ocsp responder project that can rely on crls(deltas preferred)?

Thanks.

[philfry]

Hi all,

Do you know of any good ocsp responder project that can rely on crls(deltas preferred)?

Correct me if I'm wrong, but an ocsp responder needs to know about revoked certificates as well as about valid ones to provide a valid answer.

[logopk]

Hi all,

Do you know of any good ocsp responder project that can rely on crls(deltas preferred)?

Correct me if I'm wrong, but an ocsp responder needs to know about revoked certificates as well as about valid ones to provide a valid answer.

That would be the point of openssl ocsp relying on the (complete) index.txt.

[maraino]

@philfry @logopk Yes, an OCSP responder requires to know about the active certificates. If your only source of information is the CRL you will need to fake the response of any other certificate and return the status GOOD for any other serial number.

And because you need to fake responses there won't be any OCSP responder available. You'll need to build it yourself, it's not hard.

[unreality]

ive completed all the changes except for the AuthorityID stuff, since i couldnt find a good way of accessing it. also its probably a mess now that i've rebased it :/

[LarsBingBong]

Good stuff @unreality - thank you!

@maraino - when this gets merged. If I'm consuming the step-ca via a Helm chart on Kubernetes. Step Certificates Helm chart will also need to be updated right?

Thanks

[logopk]

@philfry @logopk Yes, an OCSP responder requires to know about the active certificates. If your only source of information is the CRL you will need to fake the response of any other certificate and return the status GOOD for any other serial number.

And because you need to fake responses there won't be any OCSP responder available. You'll need to build it yourself, it's not hard.

@maraino OK, I absolutely agree. I'm sorry but you got me confused with the comment:

And with this PR merged, you will be also able to create an OCSP responder using the generated CRL as the source.

That's why I asked. Maybe you can explain this in detail?

Thank you.

[larssb]

Almost there - any day now 👍 .... the hold up is the minor stuff in this PR right? @unreality do you kindly know?

[LarsBingBong]

Any news here @unreality and or @maraino - thank you very much 👍🏿

[unreality]

Ive brought this branch up to latest master, @maraino could you review and let me know any changes that are needed?

[LarsBingBong]

@maraino a polite request for a PR review 👍🏿 - are we there now?

Thanks

[maraino]

@maraino a polite request for a PR review 👍🏿 - are we there now?

Thanks

I'm on vacation right now. It will have to wait until I get back.

[LecrisUT]

@unreality A few things:

• Could you add appropriate flags to step ca init to enable this in the ca.json generation
• Clarify and/or add the options to add the CRL to either or both the root and intermediate certificates. For example there could be a school root certificate with its own CRL to invalidate intermediates, and lab intermediate CRL to invalidate their own managed services
• A few simple words in the docs section to make it known

[maraino]

@unreality @LecrisUT I think I will have time this week to look into this PR. But there's one thing to consider: we don't see this as the final implementation, and it might go away at some point once authority webhooks are baked into step-ca.

Those webhooks will act on different certificate events. For example, when a certificate is issued, you will be able to configure a webhook that will send the issued certificate (or perhaps a pre-certificate) to a third party, so it can store it or perhaps, in some cases, perform some validation, the exact details are still not designed. In the same way, when a certificate is revoked, we will send the serial and the certificate information we have to a third party that can use this to generate the CRLs, and store them in a bucket, so it's accessible to the users.

If we implement any of those third-party services is not yet decided, but the revocation one is probably the best candidate.